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Abstract 

Perfonnance  metrics  have  helped  to  sustain  the  Air  Force,  improve  processes,  and 
guided  decisions  makers  through  decades  of  challenges  and  change.  The  Air  Force 
continues  to  change  as  it  faces  the  challenges  of  an  aging  fleet  coupled  with  the  tightest 
budget  constraints  of  modem  times.  The  current  metrics  employed  by  the  United  States 
Air  Force  Aircraft  Maintenance  community  have  gone  largely  unchanged  over  decades 
despite  a  host  of  force  altering  events.  The  focus  of  this  research  is  to  evaluate  current 
maintenance  metrics  and  assess  the  utility  of  the  Balanced  Scorecard  framework  for  use 
in  a  Maintenance  Group.  The  researcher  utilizes  a  mixed  methodology  to  accomplish  this 
evaluation,  including  survey  research,  statistical  analysis,  content  analysis,  and 
correlation  analysis.  The  paper  proposes  a  Maintenance  Group  Balanced  Scorecard 
based  on  the  analysis  of  survey  responses  from  Maintenance  Officers  with  Combat  Air 
Forces  (CAF)  experience.  The  proposed  Balanced  Scorecard  is  comprised  of  existing, 
refined,  and  proposed  metrics  to  measure  each  perspective  category  of  the  Balanced 
Scorecard,  and  is  intended  to  help  align  maintenance  metrics  with  organizational 
goals/objectives  and  the  strategic  goals  of  Maintenance  Groups  in  CAF  units. 
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A  Survey  and  Analysis  of  Aircraft  Maintenance  Metrics: 
A  Balanced  Scorecard  Approach 


I.  Introduction 


General  Issue 

Successful  organizations  have  long  recognized  perfonnance  metrics  as  a  vital 
method  of  establishing  easily  understood  and  tangible  goals  for  employees, 
understanding  where  processes  are  succeeding,  or  identifying  areas  in  need  of 
improvement.  Perfonnance  metrics  provide  leaders  and  management  personnel  a  means 
to  manage  by  fact  rather  than  by  feel  or  anecdotal  infonnation.  Well-developed  metrics 
lend  themselves  to  objective  decision  making  and  provide  decision  makers  statistical 
proof  to  validate  decisions  already  made  or  to  guide  them  into  the  future.  Additionally, 
the  study  of  objective  performance  metrics  solidifies  constancy  of  purpose  for  every 
person  vested  in  the  success  of  the  organization  and  propels  the  organization  on  the 
desired  course.  Since  the  goal  of  any  manager  is  to  extract  the  maximum  potential  of 
every  resource  in  every  process,  effective  measurement  tools  are  necessary  to  increase 
efficiency  and  value,  and  minimize  waste  and  error.  In  this  manner,  comprehensive 
perfonnance  measurement  serves  to  focus  attention  and  resources  toward  successful 
organizational  behavior  and  minimizes  the  time  and  effort  spent  on  identifying  those 
behaviors  that  require  modification  to  contribute  more  effectively  to  achieve 
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organizational  goals.  Quite  simply,  organizations  develop  strategy  that  is  refined  through 
goals,  and  progress  towards  meeting  goals  is  measured  using  metrics. 

Metrics  are  measurements  that  can  be  described  as  a  simple  count  of  actions  or 
events  over  time,  a  ratio  of  one  value  to  another,  or  a  complex  relationship  among 
organizations,  people,  objects,  processes  or  events.  Perfonnance  measurements  are 
specific  types  of  metrics  and  are  described  as,  “The  tools  we  use  to  determine  whether  we 
are  meeting  our  objectives  and  moving  toward  the  successful  implementation  of  our 
strategy.  Specifically,  we  may  describe  measures  as  quantifiable  standards  used  to 
evaluate  and  communicate  perfonnance  against  expected  results”  (Niven,  2002,  p.  1 14). 
The  tenninology  of  metrics,  maintenance  metrics,  and  perfonnance  measurements  are 
used  interchangeably  in  the  context  of  this  thesis,  as  all  are  tools  for  leaders  or  managers 
to  assess  actions  and  processes  in  their  organization,  and  to  assist  in  decision-making. 
“Ultimately,  the  actions  people  take  and  the  decisions  they  make  detennine  the  degree 
and  nature  of  value  that  an  operation  creates.  These  actions  and  decisions  can  be  greatly 
influenced  by  metrics”  (Melnyk,  et  ah,  2004,  p.  211). 

There  are  fewer  organizations  in  the  Air  Force  where  decisions  are  more  critical 
to  safe  and  successful  mission  accomplishment  than  in  combat  aviation  aircraft 
maintenance.  Aircraft  maintenance  organizations  exist  in  a  highly  dynamic  environment 
with  dwindling  resources  and  priorities  that  shift  almost  daily.  These  factors  and  the 
importance  of  the  mission  necessitate  effective  performance  metrics  to  maximize 
readiness  and  combat  capability.  Commanders  and  decision  makers  require  fidelity  in 
infonnation  presented  in  the  most  easily  understood  format  to  facilitate  critical  decisions 
in  a  time  sensitive  environment.  It  is  paramount  that  perfonnance  metrics  be  easily 
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understood,  timely,  accurate,  fact-based  and  immune  to  manipulation,  and  relevant  to 
challenges  of  the  changing  environment. 

Background 

From  the  beginning  of  aviation— the  establishment  of  the  Aeronautical  Division  of 
the  Anny  Signal  Corps  in  August  1907  and  the  first  flight  of  Wright  Brother’s  flyer  in 
September  1907— came  the  inception  of  the  aircraft  mechanic  who  was  separate  from  the 
pilot  who  flew  the  aircraft.  The  specialized  field  of  aircraft  maintenance  was  formally 
established  in  1911,  with  the  publication  of  Provisional  Airplane  Regulations  for  the 
Signal  Corps,  United  States  Army,  1911.  Even  during  the  infancy  of  the  Air  Force, 
leaders  and  decision-makers  recognized  the  necessity  for  reporting  infonnation  for 
decision  making  and  understanding  of  perfonnance  of  aircraft— the  primary  fonnal 
metrics  reported  by  flying  units  to  higher  command  levels  were  in-commission  or  out-of¬ 
commission  rates  and  accident  rates  (Townsend,  1978). 

In  September  of  1956,  the  Air  Force  codified  the  need  for  fonnal  performance 
measurement,  reporting,  and  analysis  when  it  published  AFM  66-1,  Maintenance 
Management,  establishing  the  first  set  of  standards,  goals,  and  objectives  for  aircraft 
maintenance.  “The  established  standards  included  aircraft  in-commission  rates, 
component  repair  standards,  and  aircraft  scheduling  objectives,  among  many  others.  This 
infonnation  told  the  maintenance  man  what  he  was  expected  to  accomplish,  and  gave  him 
the  capability  to  measure  his  effectiveness  in  meeting  these  standards  or  goals” 
(Townsend,  1978). 

From  1956  to  current  day,  the  Air  Force  has  continued  to  use  metrics  in  the 
Aircraft  Maintenance  community  for  “The  measurement  of  the  many  logistics  processes 
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that  provide  combat  capability  to  the  unit. . .  [and]  quick  and  accurate  identification  of 
areas  for  improvement,  as  well  as  identification  of  support  problems  beyond  the  scope  of 
the  unit”  (Air  Combat  Command  Instruction  21-1 18,  2012,  p.  4).  The  list  of  specific 
sanctioned  and  reportable  metrics  evolved  and  expanded  over  time  through  multiple 
iterations  of  aircraft  maintenance  management  policies  and  regulations  such  as  Air  Force 
Manual  66-1  (1972)  and  TAC  Regulation  66-3;  however,  since  the  2003  release  of  Air 
Force  Policy  Directive  21-1,  Maintenance,  Air  and  Space  Maintenance,  the  list  of 
sanctioned  metrics  has  remained  largely  unchanged. 

The  mission  of  the  United  States  Air  Force  is  among  the  most  complex  in  the 
world.  With  installations,  aircraft  of  various  mission  design  and  purpose,  equipment  and 
people  in  every  corner  of  the  world  engaged  in  a  seemingly  endless  number  of  on-going 
missions,  it  is  vitally  important  that  leaders  understand  the  state  of  their  organization’s 
operations  to  the  furthest  extent  possible.  This  is  of  particular  importance  in  the  aircraft 
maintenance  arena. 

The  overarching  objective  of  AF  maintenance  is  to  maintain  aircraft  and 
equipment  in  a  safe,  serviceable  and  ready  condition  to  meet  mission  needs. 
Maintenance  management  metrics  serve  this  overarching  objective  and  are 
established  and  maintained  by  Headquarters  Air  Force  (HAF),  Major  Commands 
(MAJCOMs),  Wings  and/or  Squadrons  to  evaluate/improve  equipment  condition, 
personnel  skills  and  long-tenn  fleet  health  (Air  Combat  Command  Instruction  21- 

1 18,  2012,  p.  28) 

Aircraft  maintenance  metrics  are  maintained  and  used  to  monitor  and  drive 
improved  performance  from  HAF  through  MAJCOMs  all  the  way  down  to  the  squadron 
level.  Furthermore,  unit  perfonnance  metrics  reach  beyond  the  internal  workings  of  the 
Air  Force  but  also  convey  infonnation  about  the  Air  Force  to  agencies,  committees  and 
people  in  positions  of  power  outside  of  the  Air  Force  that  have  significant  influence. 
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Communication  with  policy  makers  at  the  Department  of  Defense  and 
Congressional  level  undoubtedly  has  far-reaching  and  lasting  impact.  It  is  imperative  the 
communication  is  fact-based  and  clearly  articulates  the  successes,  challenges,  direction 
and  needs  of  the  Air  Force.  The  Air  Force  uses  metrics  to  produce  statistical  analysis  for 
congressional  committees,  the  Office  of  Management  and  Budget,  and  the  Department  of 
Defense.  Lead  MAJCOMs  establish  capability  goals  in  coordination  with  the  Air  Staff. 
These  goals  enable  HAF  to  assess  resource  allocation  funding  on  a  quarterly  basis,  and 
go  into  the  yearly  Readiness  Reports  to  Congress.  The  Air  Force  uses  metrics  reports  to 
develop  and  defend  the  US  Air  Force  input  to  the  Planning,  Programming,  Budgeting, 
and  Execution  (Department  of  the  Air  Force  Instruction  21-103,  2012)  for  decisions  on 
issues  such  as  airframe  retirements,  airframe  acquisition,  manpower  increases  and 
reductions,  budget  increases  or  decreases,  or  suitability  of  units  for  participation  in 
conflicts  (Durand,  2008). 

The  Air  Force  has  also  used  metrics  to  aid  in  major  organizational  restructuring 
decisions  for  aircraft  maintenance  and  operations.  The  decision  to  transition  from 
Production  Oriented  Maintenance  Organization  to  the  Combat  Oriented  Maintenance 
Organization  in  1978  and  the  transition  of  the  Objective  Wing  Organization  to  the 
Combat  Wing  Organization  in  2002  were  both  made  as  an  attempt  to  reverse  declining 
maintenance  performance  metrics  (Durand,  2008). 

Useful  and  accurate  perfonnance  measurement  metrics  have  helped  to  sustain  the 
Air  Force,  improve  processes,  and  guided  decisions  makers  through  decades  of 
challenges  and  change.  An  aging  fleet,  sequestration  and  subsequent  budget  constraints 
are  the  newest  of  the  challenges  the  Air  Forces  faces.  Maj.  Gen.  Edward  L.  Bolton  Jr,  the 
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Deputy  Assistant  Secretary  for  Budget,  Office  of  the  Assistant  Secretary  of  the  Air  Force 

for  Financial  Management  and  Comptroller  described  these  challenges: 

In  terms  of  average  aircraft  age,  Air  Force  “iron”  is  older  than  it  has  ever  been. 
Additionally,  high  operations  tempo  has  shortened  service  lives,  increasing  the 
cost  to  sustain  and  maintain  our  weapon  systems.  Faced  with  compounding  fiscal 
challenges,  we  must  make  prudent  choices  to  ensure  the  Air  Force  continues  to 
preserve  our  Nation’s  airpower  advantage.  To  this  end,  the  Air  Force  is 
committed  to  avoiding  a  hollow  force;  one  that  looks  good  on  paper,  but  has  more 
units  and  equipment  than  it  can  support,  lacks  the  resources  to  adequately  man, 
train  and  maintain  them,  or  keep  up  with  advancing  technologies. 

With  these  changes,  the  active  duty  Air  Force  will  reduce  to  approximately 
329,500  personnel  in  FY  2013,  approaching  the  same  size  as  when  we  were 
established  as  a  separate  service  in  1947.  At  the  same  time,  the  FY  2013  NDAA 
pennits  the  Air  Force  to  proceed  with  selected  aircraft  retirements  and  transfers 
necessary  to  meet  budget  targets  while  protecting  readiness  and  modernization. 

With  the  onset  of  Sequestration,  we  have  begun  implementing  immediate  actions 
to  mitigate  an  approximate  $10B  reduction  to  Air  Force  Total  Obligation 
Authority.  We’ve  taken  steps  to  minimize  impacts  to  readiness  and  our  people; 
however,  the  results  of  these  cuts  will  be  felt  across  all  Air  Force  Core  Missions 
and  challenge  the  goals  of  our  FY  2014  Budget  Submission  which  does  not  reflect 
Sequestration  reductions.  Given  today’s  fiscally  constrained  environment,  the  Air 
Force  must  pursue  the  best  combination  of  choices  available  to  balance  force 
reductions  and  manage  war-fighting  risks,  resources  and  the  bow-wave  of  impacts 
from  FY  2013.  Taking  these  actions  allows  us  to  keep  faith  with  our  687,634 
total  force  Ainnen  and  continue  to  excel  in  our  role  to  fly,  fight,  and  win  in  air, 
space  and  cyberspace  (Bolton,  2013,  p.  4). 


Clearly,  Air  Force  resources  will  require  more  attention  and  more  critical 
decisions  than  ever  before.  Couple  the  challenge  of  the  aging  fleet,  the  effort  to 
recapitalize,  with  the  a  reduced  force  structure  during  the  tightest  budget  constraints  of 
modem  times  and  one  can  easily  see  how  maintenance  management  metrics  will  play  an 
increasingly  important  role.  The  importance  of  a  sound  aircraft  maintenance  metrics  set 
is  more  critical  than  ever  before,  and  the  evaluation  of  the  current  metrics  set  is 
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imperative  to  ensure  aircraft  maintenance  metrics  are  meeting  the  needs  of  Air  Force 
decision  makers  at  every  level. 

Problem 

Air  Policy  Directive  21-1,  Maintenance,  Air  and  Space  Maintenance  provides 
direction  from  the  departmental  level  down  to  the  wings  for  reporting  performance 
measurements  but  hasn’t  been  reviewed  or  changed  since  February  2003;  however,  the 
Air  Force’s,  and  the  Maintenance  Group’s  situation  has  changed  vastly  since  that  time.  A 
review  of  the  maintenance  metrics  is  needed  to  ensure  it  is  relevant,  and  that  it  is  meeting 
the  needs  of  the  maintainers  that  use  it.  “There  is  a  pressing  need  for  companies  to 
reevaluate  their  perfonnance  measurement  systems.  This  reevaluation  should  be 
conducted  for  both  the  individual  metrics  and  the  perfonnance  measurement  system  as  a 
whole”  (Caplice  &  Sheffi,  1994,  p.  1 1).  No  MAJCOM-wide  study  has  been  accomplished 
to  detennine  whether  the  current  set  of  sanctioned  metrics  is  useful  for  the  leaders  and 
decision-makers  that  use  them,  or  if  metrics  are  in  need  of  revision,  addition  or  deletion. 

Additionally,  metrics  traditionally  tracked  in  a  Maintenance  Group  are  not 
mandated  to  be  studied  in  an  integrated  fashion.  Some  metrics  are  mandated  to  be 
studied  and  discussed  in  a  group  setting,  while  others  are  only  required  to  be  reported  to 
Higher  Headquarters  (Air  Combat  Command  Instruction  21-118,  2012).  Many  separate 
meetings  take  place  where  a  limited  number  and  type  of  metrics  are  discussed,  such  as 
Maintenance  Status  of  Training  and  Maintenance  Standardization  and  Evaluation 
Program,  but  no  meeting  is  mandated  which  integrates  all  metrics  for  a  consolidated  view 
of  the  performance  of  all  facets  of  successful  perfonnance.  The  limited  scope  of  these 
individual  meetings  provides  a  relatively  myopic  view  of  the  entire  wing  and  limits  the 
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ability  of  leaders  and  managers  to  see  the  interrelation  between  the  various  processes  that 
contribute  to  the  end  goal  of  the  organization.  Studying  different  types  of  metrics  at 
different  times  makes  it  difficult  to,  “see  whether  improvement  in  one  area  may  have 
been  achieved  at  the  expense  of  another”  (Kaplan  &  Norton,  1992,  p.  73).  Lack  of 
standardization  further  limits  the  effectiveness  of  these  compartmentalized  meetings. 

The  frequency  and  interval  of  meetings  to  study  and  analyze  metrics  is  not  standardized, 
and  neither  are  the  required  attendees  at  these  established  meetings. 

Research  Objectives 

The  objective  of  this  research  was  to  evaluate  individual  maintenance  metrics  and 
the  sanctioned  maintenance  metrics  set  currently  employed  by  the  Aircraft  Maintenance 
community.  The  researcher  sought  to  assess  the  relevance  of  current  metrics  to  the 
leaders  and  decision  makers  that  use  them,  as  well  as  the  optimal  frequency  for  their 
review.  In  an  effort  to  evaluate  the  efficacy  of  individual  metrics  for  inclusion  into 
selected  framework,  the  researcher  sought  to  evaluate  the  metrics  against  an  objective 
evaluation  criteria  summarized  from  previous  research.  Lastly,  to  better  align 
maintenance  metrics  with  organizational  goals/objectives  and  the  strategic  goals  of  the 
Air  Force,  the  researcher  sought  to  adapt,  propose  and  evaluate  a  balanced  scorecard 
metrics  framework  for  use  by  Maintenance  Group  leaders  and  decision  makers. 

Research  Question 

1 .  What  metrics  should  be  used  in  the  Maintenance  Group  Balanced  Scorecard 
framework  to  meet  the  needs  of  the  CAF  Maintenance  Community? 


II.  Literature  Review 


The  general  intent  of  this  literature  review  was  to  study  a  broad  spectrum  of 
publications  across  different  contexts  in  order  to  understand  the  importance  of  metrics  as 
a  critical  managerial  challenge,  and  to  gain  insights  into  various  approaches  to,  and 
perspectives  of,  metrics  and  their  requirements  to  support  the  decision  making  process. 
Specifically,  the  researcher’s  objectives  during  this  literature  review  were  to  determine 
effective  criteria  for  evaluating  individual  metrics,  and  gain  a  thorough  understanding  of 
the  balanced  scorecard  framework  to  apply  to  USAF  Combat  Air  Force  (CAF) 
maintenance  operations.  In  order  to  achieve  these  objectives,  the  researcher  classified 
and  analyzed  the  literature  in  several  different  ways  (Appendix  A). 

The  first  way  the  researcher  classified  publications  was  by  the  origin  of 
publication,  from  either  the  DoD  or  USAF  sector  or  the  private  industry  or  academic 
sector.  The  purpose  of  researching  literature  from  both  origins  was  first,  to  establish  an 
understanding  of  metrics-related  research  accomplished  supporting  functions  unique  to 
the  DoD  or  USAF;  second,  to  collect  the  current  metrics  used  to  measure  performance  in 
the  field;  and  third,  to  understand  themes  and  perspectives  of  both  sectors  related  to 
perfonnance  measurement  identification,  design,  and  characteristics. 

The  researcher  further  classified  the  literature  by  identifying  general  focus  areas 
of  each  publication,  which  examined  or  addressed  metrics  concepts  from  broad 
conceptual  models  or  frameworks,  to  more  refined  metrics  systems  or  sets,  to  a  very 
focused  few  or  individual  metrics. 
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The  third  classification  the  researcher  explored  was  the  focus  of  the  topic 
categories  each  article  addressed.  The  researcher  sought  to  include  literature  across  six 
topic  categories  to  facilitate  a  well-developed  knowledge  base  to  support  this  research. 
The  topic  categories  included  literature  that  addressed: 

1 .  The  purpose,  importance,  or  characteristics  of  its  respective  focus  area 

2.  Proposals  of  specific  metrics,  metrics  sets/systems,  or  frameworks 

3.  The  design,  development,  or  selection  of  one  of  more  of  the  focus  areas 

4.  The  implementation  of  individual  metrics,  systems  or  frameworks 

5.  The  management  of  metrics,  metrics  sets,  or  conceptual  frameworks 

6.  The  evaluation  of  one  of  more  of  the  focus  areas 

Lastly,  the  researcher  sought  to  classify  the  publications  by  the  methodology  used 
to  conduct  the  research.  The  publications  were  assembled  into: 

1 .  Literature  Review/Content  Analysis 

2.  Case  Study-based 

3.  Survey/interview-based 

4.  Conceptual  Model/Framework-based 

5.  Expert  opinion 

6.  Mathematical/statistical  analysis 
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Of  particular  interest  to  the  researcher  were  three  Air  Force  publications  related  to 
metrics  in  the  Maintenance  Group,  which  served  as  foundations  for  the  researcher’s 
methodology.  The  first  was  Capt  Brian  Waller’s  thesis  Evaluation  of  Air  Force  Aircraft 
Maintenance  Metrics  for  Integration  into  the  Expeditionary  Combat  Support  System.  In 
his  thesis,  Capt  Waller  recommended  expanding  his  case  study  of  suitable  and  useful 
metrics  to  be  incorporated  into  the  Expeditionary  Combat  Support  System  into  a 
statistical  survey  by  a  larger  population  of  maintenance  experts.  He  states,  “Further 
revelations  may  be  developed  by  expanding  the  expert  pool  to  include  maintenance 
managers  from  other  organizations,  such  as  fighter  or  special  operations  units  as  well  as 
evaluations  from  managers  at  higher-level  headquarters”  (Waller,  2009,  p.  115). 
Additionally,  he  determined,  “The  results  of  this  study  have  found  a  number  of  metrics 
that  need  re-evaluation,  if  not  re-engineering.  Further  exploration  should  be  undertaken 
in  order  to  determine  how  to  best  change  these  metrics  to  align  better  with  the  strategic 
goals  of  the  Air  Force  as  well  as  the  goals  for  effective  metrics”  (Waller,  2009,  p.  1 15). 

The  second  study  was  a  report  written  by  Capt  Emily  Harris  titled  Development  of 
Aerospace  Ground  Equipment  (AGE)  Metrics,  which  addressed  the  lack  of  standardized 
metrics  to  measure  the  perfonnance  of  AGE  maintenance,  and  proposed  a  set  of  metrics 
to  be  used  at  the  Maintenance  Group  level  (Harris,  2011). 

Lastly,  the  Air  Force  Research  Laboratory  and  the  University  of  Arkansas 
published  a  report  called  The  Use  of  Decision  Models  in  the  Development  of  a 
Collaborative  Integrated  Solutions  System  which  identified  a  need  to  have  a  strategically 
aligned  performance  measurement  system  for  flightline  maintenance  activities,  and  used 
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the  Kaplan  and  Norton’s  Balanced  Scorecard  approach  to  develop  a  proposed  metrics  set 
for  use  by  an  Aircraft  Maintenance  Unit  (Nachtmann,  et  al.,  2003). 

Appendix  A  shows  the  researcher’s  breakdown  of  all  reviewed  literature  and  the 
detennined  categorizations. 

Officially  Sanctioned  Metrics 

The  next  objective  of  the  literature  review  was  to  detennine  the  officially 
sanctioned  metrics  in  aircraft  maintenance  through  a  literature  review  of  regulations  and 
Air  Force  Instructions  (AFI).  The  researcher  found  that  metrics  were  scattered  across 
many  different  aircraft  maintenance  and  training  regulations,  supplements,  and  Tactics, 
Techniques  and  Procedures,  and  identified  over  a  hundred  metrics  currently  in  use.  The 
researcher  called  this  list  of  unfiltered  and  unconsolidated  metrics  “Exhaustive  List  of 
Metrics”  which  is  listed  in  Appendix  B.  For  the  ease  of  surveying  the  maintenance 
officer  population,  the  researcher  consolidated  metrics  that  she  deemed  similar  in  intent 
to  make  a  “Consolidated  List  of  Metrics”,  which  can  be  found  in  Appendix  C. 

Current  Frequency  and  Levels  of  Metrics  Study 

After  researching  and  determining  the  officially  sanctioned  metrics,  the  researcher 
sought  to  determine  the  mandated  frequency  and  management  level  of  study  of  the 
officially  sanctioned  metrics  through  a  literature  review  of  regulations  and  AFIs.  The 
researcher  noted  that  the  set  of  “traditional”  aircraft  maintenance  metrics  actually  had  no 
AFI  mandated  meeting  at  the  Wing  or  Maintenance  Group  level,  while  other  metrics  had 
AFI  mandated  meetings  for  their  study  by  leaders  and  managers.  The  researcher  outlined 
the  frequency  and  level  of  management  in  which  the  officially  sanctioned  metrics  are 
reviewed  in  Appendix  D. 
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Evaluation  Criteria  for  Individual  Metrics 


To  aid  in  detennining  the  efficacy  of  the  individual  metrics,  the  literature  review 
included  research  to  establish  a  viable  set  of  objective  criteria  for  evaluating  individual 
metrics.  After  carefully  considering  various  methods  of  evaluating  individual  metrics 
discovered  in  her  exploration,  the  researcher  determined  the  Caplice  and  Sheffi  criteria 
was  the  most  comprehensive  and  complete  set  of  criteria  to  use  in  her  study.  In  the 
literature  review  performed  by  Caplice  and  Sheffi,  they  evaluated  past  publications  on 
metrics  evaluation,  and  found  several  common  general  characteristics  in  good 
perfonnance  metrics  (Caplice  &  Sheffi,  1994).  They  produced  a  set  of  eight  evaluation 
criteria  based  on  this  review,  which  are  described  below  and  outlined  in  Figure  1 . 

Validity 

“A  metric  is  valid  if  it  reflects  the  actual  activity  being  performed  and  controls  for 
any  exogenous  factors  that  are  out  of  the  process  manager's  control”  (Caplice  &  Sheffi, 
1994,  p.  15).  For  example,  Deviation  Rate  may  be  considered  valid  because  it  accurately 
measures  departures  from  the  printed  flying  schedule,  takes  into  account  external  factors 
such  as  weather. 

Robustness 

“A  metric  is  robust  if  it  is  widely  accepted,  is  interpreted  similarly  by  different 
users,  and  can  be  used  for  comparisons  across  time,  locations,  and  organizations” 
(Caplice  &  Sheffi,  1994,  p.  15).  For  example,  Fully  Mission  Capable  rate  may  be 
considered  robust  because  it  is  interpreted  the  same  by  all  that  use  it,  is  measured  the 
same  way  at  any  time  or  at  any  location  by  all  organizations  and  is  easily  repeatable. 
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Usefulness 

“A  metric  is  useful  if  it  is  readily  understood  by  the  decision  maker  and  suggests  a 
course  of  action  or  direction  to  be  taken”  (Caplice  &  Sheffi,  1994,  p.  15).  For  example, 
Aircraft  Availability  may  be  considered  useful  because  the  decision  maker  can 
immediately  understand  the  limitation  imposed  by  a  low  number  of  available  aircraft.  He 
may  take  action  to  reduce  the  number  of  aircraft  employed  for  non-mission  priority 
purposes  such  as  trainers. 

Integration 

“A  metric  is  integrative  if  it  incorporates  all  of  the  major  components  and  aspects 
of  the  process  being  measured  and  promotes  coordination  across  functions,  divisions,  or 
firms  in  the  supply  chain.  The  primary  thrust  of  this  criterion  is  to  promote  coordination 
between  the  players  involved  in  the  process”  (Caplice  &  Sheffi,  1994,  p.  15).  For 
example,  Total  Not  Mission  Capable  rate  may  be  considered  integrated  because  it 
includes  both  maintenance  and  supply  functions  and  encourages  coordination  between 
maintenance  and  supply  agencies  to  reduce  the  number  of  Not  Mission  Capable  aircraft 
(Caplice  &  Sheffi,  1994). 

Economy 

“A  metric  is  economical  if  the  benefit  of  tracking  it  outweighs  the  cost  to  collect, 
process,  and  report  it.  This  is  more  of  a  judgment  call  than  a  strict  cost-benefit 
comparison  so  that  the  economy  criterion  should  be  used  to  select  between  potential 
metrics  rather  than  for  the  decision  of  whether  to  use  any  metric  at  all”  (Caplice  &  Sheffi, 
1994,  p.  15).  For  example,  Mission  Impaired  Capability  Awaiting  Parts  may  be 
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considered  economical  because  it  is  relatively  easy  to  collect  the  data  and  the  benefit  of 
putting  higher  priority  on  parts  returns  aircraft  to  service  faster. 

Compatibility 

“A  metric  is  compatible  with  the  existing  data  collection,  infonnation  systems, 
and  information  flows  of  the  firm  if  no  significant  additional  work  is  required  to  install 
and  use  it.  While  compatibility  has  some  overlap  with  the  economy  criterion,  in  that  any 
system  can  be  made  to  be  compatible  to  a  proposed  metric  given  the  needed  time  and 
money,  they  are  not  the  same.  A  metric  which  is  economical  in  terms  of  collecting  and 
reporting  data  might  not  always  be  compatible  with  the  existing  flow  of  information” 
(Caplice  &  Sheffi,  1994,  pp.  15-16).  For  example,  Average  Sortie  Duration  may  be 
considered  compatible  because  the  information  is  collected  during  an  existing  process 
(Pilot  debrief)  and  captured  in  an  already  existing  database. 

Level  of  Detail 

“A  metric  has  the  correct  level  of  detail  if  it  captures  and  reports  the  data  in  a 
level  of  aggregation  or  granularity  to  be  useful  to  the  decision  maker”  (Caplice  &  Sheffi, 
1994,  p.  16)  .  For  example,  UTE  rate  may  be  considered  to  have  the  proper  level  of 
detail  because  it  is  measured  monthly  (daily  or  weekly  is  too  often)  and  gives  the 
decision  maker  an  idea  of  how  often  aircraft  are  being  flown  during  a  30  day  period. 

Behavioral  Soundness 

“A  metric  that  is  behaviorally  sound  discourages  any  counter-productive  actions 
or  game-playing  by  those  process  owners  or  organizations  being  measured.  While  it  is 
always  hoped  that  a  measure  will  align  peoples'  actions  with  the  organization's  overall 
objectives,  in  many  cases  it  can  provide  incentives  for  doing  the  opposite”  (Caplice  & 
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Sheffi,  1994,  p.  16).  For  example,  Abort  rate  may  be  considered  behaviorally  sound 
because  it  is  simply  a  report  of  an  event  and  there  are  no  counterproductive  actions  or 
means  to  “game  the  system”  to  improve  the  metric,  but  it  is  a  top  indicator  of  how  well 
maintenance  executes  the  flying  mission. 


Criterion 

Description 

Validity 

The  metric  accurately  captures  the  events  and  activities  being 
measured  and  controls  for  any  exogenous  factors 

Robustness 

The  metric  is  interpreted  similarly  by  the  users,  is  comparable  across 
time,  location,  and  organizations.  Metric  is  repeatable. 

Usefulness 

The  metric  is  readily  understandable  by  the  decision  maker  and 
provides  a  guide  for  action  to  be  taken 

Integration 

The  metric  includes  all  relevant  aspects  of  the  process  and  promotes 
coordination  across  function  and  divisions. 

Economy 

The  benefits  of  using  the  metric  outweigh  the  cost  of  data  collection, 
analysis,  and  reporting 

Compatibility 

The  metric  is  compatible  with  the  existing  infonnation,  material,  and 
cash  flows  and  systems  in  the  organization 

Level  of  Detail 

The  metric  provides  a  sufficient  degree  of  granularity  or  aggregation 
for  the  user 

Behavioral 

Soundness 

The  metrics  minimized  incentive  for  counterproductive  acts  or  game 
playing  and  is  presented  in  a  useful  fonn 

Figure  1:  Metric  Evaluation  Criteria  (Caplice  &  Sheffi,  1994,  p.  14) 


Metrics  Set  Theory/Framework 

The  researcher  included  an  examination  of  established  theory  and  frameworks  on 
how  metrics  sets  are  constructed  in  the  literature  review.  The  purpose  of  this 
examination  was  to  solidify  understanding  of  the  various  theories  and  approaches  to 
metrics  set  construction  and  to  select  a  theory  to  build  the  Maintenance  Group’s  metric 
set  against.  The  researcher  chose  to  scope  this  research  around  the  balanced  scorecard 
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framework  from  the  many  available  because  of  the  credibility  of  the  source  and  because 
of  its  enduring  success  in  corporate  applications. 

Balanced  Scorecard  overview 

The  balanced  scorecard  approach  to  constructing  effective  metrics  sets  was 

introduced  in  an  attempt  to  reconcile  performance  measurement  problems  in  traditional 

management  strategies.  The  creators  of  the  balanced  scorecard  framework,  Kaplan  and 

Norton,  realized  the  vital  importance  of  having  an  easily  understood,  yet  comprehensive 

method  of  providing  top  managers  sufficient  details  on  operational  measures  regarding 

financial  measures,  results  of  decisions,  customer  satisfaction,  internal  processes  and 

innovation  and  improvement  activities.  According  to  authors  Kaplan  and  Norton,  the 

balanced  scorecard  was  developed  as  a  set  of  measures  used  to  give  managers  an  all- 

inclusive  view  of  their  organization’s  performance: 

The  balanced  scorecard  includes  financial  measures  that  tell  the  results  of  actions 
already  taken,  and  compliments  the  financial  measures  with  operational  measures 
on  customer  satisfaction,  internal  process,  and  the  organization’s  innovation  and 
improvement  activities — operational  measures  that  are  the  drivers  of  future 
financial  performance  (Kaplan  &  Norton,  1992,  p.  71). 

During  the  development  of  the  balanced  scorecard  theory,  Kaplan  and  Norton 

recognized  that  for  an  organization  to  successfully  achieve  its  objectives,  a  clear  strategy 

is  necessary  to  ensure  there  is  a  balanced  approach  to  avoid  compromising  one  goal  for 

another  and  thus  jeopardizing  the  entire  endeavor.  One  of  the  most  prevalent  failures  in 

management  strategy  is  failing  to  identify  the  interrelation  of  the  various  processes  and 

aspects  in  the  organization.  As  stated  in  the  Balanced  Scorecard  Institute’s  website: 

Traditional  management  strategies  overemphasized  financial  measures  at  the 
expense  of  progress  and  growth.  This  overemphasis  brought  about  short-term 
gains  to  the  detriment  of  long-term  success.  The  balanced  scorecard  is  a 
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performance  management  system  that  allows  organizations  to  clarify  their 
strategy  and  assure  that  every  aspect  of  operations  is  directed  toward  the  success 
of  these  goals”  (Balanced  Scorecard  Institute,  1998-2013). 

Successful  strategists  also  understand  that  less  traditional,  non-financial  perfonnance 

measures  add  value  to  a  successful  organizational  strategy.  In  addition  to  the  traditional 

financial  performance  measures,  the  balanced  scorecard  incorporates  non-financial 

measures  that  create  value  for  an  organization  such  as  customer  relationships,  skills  and 

knowledge  of  the  workforce,  and  technology. 

In  order  for  an  organization  to  achieve  its  goals,  its  perfonnance  measurement 

system  must  align  with  the  overall  mission  of  the  organization.  The  four  perspectives  of 

the  balanced  scorecard  focus  on  the  mission  or  strategic  objectives  of  an  organization  and 

include  the  Customer  Perspective,  the  Internal  Process  Perspective,  the  Financial 

Perspective  and  the  Learning  and  Growth  Perspective.  Kaplan  and  Norton  explain  the 

purpose  of  the  four  different  perspectives:  “[Perspectives]  allow  managers  to. .  .answer 

four  basic  questions:  How  do  customers  see  us?  What  must  we  excel  at?  Can  we 

continue  to  improve  and  create  value?  How  do  we  look  to  shareholders?”  (Kaplan  & 

Norton,  1992,  p.  72)  Purposeful  and  valid  answers  to  these  questions  provide  focus  for 

the  time,  effort  and  money  invested  into  every  activity  in  every  process.  To  enhance 

understanding  of  the  four  perspectives,  refer  to  Figure  2. 
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GROWTH 

To  achieve  our  goals,  what 
___mustwe_im£rove?__ 


Figure  2:  Graphical  Representation  of  Balanced  Scorecard  (Kaplan  and  Norton,  1996) 


The  balanced  scorecard  provides  a  multi-faceted  view  of  the  organization  from 

multiple  angles  from  the  inside  out  and  from  the  outside  in.  The  report  is  a  composite 

view  of  customer  focus,  internal  process  focus,  financial  focus  and  improvement/growth 

focus.  It  reaches  beyond  being  a  snapshot  of  top-level  information  and  brings  a  critical 

evaluation  of  essential  components  of  an  organization’s  strategy. 

The  scorecard  brings  together,  in  a  single  management  report,  many  of  the 
seemingly  disparate  elements  of  a  company’s  competitive  agenda:  becoming 
customer  oriented,  shortening  response  time,  improving  quality,  emphasizing 
teamwork,  reducing  new  product  launch  times,  and  managing  for  the  long  term. 
The  scorecard  also  guards  against  sub-optimization.  By  forcing  senior  managers 
to  consider  all  the  important  operational  measures  together,  the  balanced 
scorecard  lets  them  see  whether  improvement  in  one  area  may  have  been  achieved 
at  the  expense  of  another  (Kaplan  &  Norton,  1992,  p.  73). 

The  balanced  scorecard  can  serve  as  an  effective  organizational  tool  providing 

executives  and  top-level  managers  a  comprehensive  view  of  the  business  and  a  means  to 

map  out  a  successful  strategy,  but  the  balanced  scorecard  approach  will  be  ineffective  if 

the  performance  measures  are  not  aligned  with  the  organizational  strategy.  Therefore, 

aligning  performance  measures  with  organizational  strategy  is  of  utmost  importance  to 

executives  and  top-level  managers.  According  to  Nachtman,  et  ah, 
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Organizational  Strategy  is  the  guiding  factor  behind  the  balanced  scorecard. 
Organizational  strategy  is  defined  as  a  set  of  long-term  goals  that,  if  successfully 
achieved,  will  revolutionize  the  way  a  unit  operates.  Without  strategic  alignment 
or  the  integration  of  this  organizational  strategy  into  the  balanced  scorecard,  a 
balanced  scorecard  is  merely  collection  of  performance  measures.  Strategic 
planning  and  alignment  to  a  given  strategy  should  be  the  top  priority  in  any 
balanced  scorecard  venture  (Nachtmann,  et  ah,  2003,  pp.  16-17). 

The  balanced  scorecard  was  introduced  as  a  valuable  method  for  private 

businesses  to  develop  successful  strategy  and  measure  perfonnance.  The  researcher 

studied  the  balanced  scorecard  and  sought  to  apply  the  principles  to  the  benefit  of  the 

mission  of  the  Air  Force,  and  more  specifically,  the  Maintenance  Groups  in  Combat  Air 

Forces  (CAF).  To  explore  its  potential  for  use,  the  researcher  reviewed  the  perspectives 

proposed  by  Kaplan  and  Norton,  and  then  considered  how  these  perspectives  could 

translate  and  apply  to  a  Maintenance  Group. 

Customer  Perspective 

The  customer  perspective,  the  first  proposed  by  Kaplan  and  Norton,  helps  to 
establish  the  fundamental  purpose  and  existence  of  an  organization.  “When  choosing 
measures  for  the  Customer  perspective  of  the  Scorecard,  organizations  must  answer  two 
critical  questions:  Who  are  our  target  customers?  What  is  our  value  proposition  in 
serving  them?  Sounds  simple  enough,  but  both  of  these  questions  offer  many  challenges 
to  organizations”  (Niven,  2002,  p.  15). 

In  the  corporate  world  where  gaining  market  share  to  garner  a  profit  is  the 
ultimate  goal,  the  customer  is  normally  well  defined  and  the  concept  seems  obvious;  if 
the  needs  of  customer  are  being  met  and  are  satisfied,  the  end  goal  of  gaining  market 
share  and  collecting  a  profit  is  achieved.  If  their  needs  are  not  met  and  the  customer  is 
not  satisfied,  market  share  shifts  to  another  company  and  revenue  is  lost.  When  applying 
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these  concepts  to  a  military  organization  such  as  the  Maintenance  Group  within  a  flying 
wing,  the  concept  of  who  the  customer  is  not  driven  by  market  share  and  garnering  a 
profit,  but  by  an  essential  and  required  combat  support  capability. 

Oxford  Dictionary  defines  customer  as  “a  person  or  thing  of  a  specified  kind  that 
one  has  to  deal  with”  (Oxford  University  Press,  2014).  Considering  this  definition,  the 
customer  of  the  Maintenance  Group  could  be  one,  or  all,  of  the  entities  in  a  flying  wing 
that  the  members  of  the  Maintenance  Group  deal  with— the  Aircrew  from  the  Operations 
Group,  the  maintainers  within  the  Maintenance  Group,  assigned  aircraft,  or  assigned 
equipment.  From  the  perspective  of  the  Maintenance  Group,  the  most  readily  identifiable 
customers  are  the  aircrews.  However,  the  Maintenance  Group  is  typically  the  largest, 
most  complex  organization  in  any  flying  wing  and  is  comprised  of  multiple  squadrons 
and  flights,  many  of  which  endeavor  to  benefit  others  within  the  Maintenance  Group 
itself.  For  instance,  an  avionics  backshop  is  focused  primarily  on  repairing  parts  to 
benefit  the  Aircraft  Maintenance  Squadron  that  strives  to  generate  airplanes.  A 
backshop ’s  direct  measure  of  success  is  repair  cycle  time  in  support  of  the  Aircraft 
Maintenance  Squadron.  Another  example  would  be  the  Military  Training  Flight  (MTF), 
whose  mission  it  is  to  train  maintainers  from  other  squadrons.  An  MTF  instructor  does 
not  directly  benefit  aircrews;  he  or  she  provides  training  and  education  to  other 
maintainers  and  squadrons  within  the  group  and  measures  success  by  the  quality  and 
quantity  of  training  accomplished.  These  are  just  a  very  few  examples  of  how 
organizations  within  the  Maintenance  Group  benefit  each  other  and  can  measure  success 
without  considering  aircrews  at  all.  The  intertwining  of  the  various  squadrons  and  flights 
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is  complex  and  meeting  the  demands  of  “internal  customers”  consumes  time  and  effort, 
but  is  an  absolute  necessity. 

One  might  not  normally  think  of  aircraft  and  equipment  as  a  “customer”  as  they 
are  objects  in  a  process,  but  aircraft  and  equipment  have  service  requirements  of  their 
own,  whether  or  not  they  are  flown  by  aircrew  or  used  by  maintainers.  Additionally,  the 
needs  of  aircraft  and  equipment  are  often  diametrically  opposed  with  the  needs  of 
aircrews  and  consume  an  enormous  amount  of  Maintenance  Group  resources.  Aircrews 
need  to  fly  to  train  and  be  proficient,  but  aircraft  and  equipment  require  out  of 
commission  time  for  maintenance  and  health.  The  pilot  requires  an  airplane  to  serve  his 
purpose  and  an  airplane  is  of  no  use  without  a  pilot.  The  Maintenance  Group’s  success 
comes  from  finding  the  critical  balance  to  ensure  the  best  interest  of  both  is  served. 
Ultimately,  mission  success  is  dependent  on  this  balance  and  the  needs  of  one  should  not 
consistently  be  given  preference  over  the  other. 

Aircrews  need  continuous  training  to  maintain  proficiency  and  to  prepare  for 
combat.  Providing  the  aircraft  for  Aircrews  often  comes  at  the  expense  to  aircraft 
combat-readiness  when  systems  degrade  or  fail  in  flight  or  on  the  ground.  Equally 
important  is  the  need  to  train  and  educate  the  maintenance  force,  who  must  maintain  a 
perfect  balance  of  qualifications  and  certifications  to  safely,  efficiently,  and  effectively 
provide  aircraft  to  pilots  in  combat  scenarios.  Equipment  must  be  available  and 
functional  to  allow  any  of  this  to  happen,  but  is  subject  to  wear  and  tear  just  like  aircraft. 
In  the  context  of  a  Maintenance  Group,  it  could  be  argued  that  success  comes  from 
effectively  serving  not  one,  but  four  customers.  Each  of  these  entities  competes  against 
the  other  for  the  use  of  resources,  but  cannot  succeed  if  any  one  of  the  others  is  failing. 
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Internal  Process  Perspective 

From  the  Internal  Process  Perspective,  the  organization  must  answer  the  question, 

“What  must  we  excel  at?”  Focusing  on  the  answers  to  that  question  helps  to  identify 

critical  processes  to  assist  in  prioritization,  and  add  focus,  direction  and  thrust  for  that 

organization.  Kaplan  and  Norton  state: 

Customer-based  measures  are  important,  but  they  must  be  translated  into 
measures  of  what  the  company  must  do  internally  to  meet  its  customer’s 
expectations.  After  all,  excellent  customer  performance  derives  from  processes, 
decisions,  and  actions  occurring  throughout  an  organization.  Managers  need  to 
focus  on  those  critical  internal  operations  that  enable  them  to  satisfy  customer 
needs.  The  internal  measures  for  the  balanced  scorecard  should  stem  from 
processes  that  have  the  greatest  impact  on  customer  satisfaction— factors  that 
affect  cycle  time,  quality,  employee  skills,  and  productivity,  for  example.  (Kaplan 
&  Norton,  1992,  p.  74). 

In  the  context  of  a  flying  wing  and  Maintenance  Group’s  potential  customers— be 
it  aircrew  of  the  Operations  Group,  maintainers,  aircraft  or  equipment— the  needs  of  each 
of  these  customers  is  to  be  combat-ready  to  meet  the  expectations  of  the  stakeholder,  the 
combatant  commander.  The  internal  processes  of  a  flying  wing,  specifically  the 
maintenance  complex,  are  diverse  and  many.  Each  one  of  these  processes  must  relate  to 
the  strategic  objective  of  the  Maintenance  Group,  and  ultimately  serve  the  needs  of  the 
customers  and  the  primary  stakeholder.  Air  Force  Policy  Directive  21-1,  Maintenance, 
outlines  general  processes  the  AF  must  excel  at  by  stating,  “The  AF  shall  support 
readiness  objectives  by  maintaining  equipment  in  optimum  condition,  assign  skilled 
personnel  necessary  to  support  expeditionary  air  forces,  and  manage  fleet  health  to  ensure 
long-term  capability  of  air  and  space  equipment”  (Department  of  the  Air  Force  Policy 
Directive  21-1,  2003,  pp.  1-2).  To  specify  and  narrow  the  scope,  this  paper  discusses  ten 
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vital  internal  processes  in  the  flying  wing  that  are  cornerstone  to  the  general  processes 
outlined  in  AFPD  21-1: 

1.  Aircrew  Mission/Training  Scheduling 

2.  Aircrew  Mission/Training  Execution 

3.  Aircraft  Flying  Scheduling 

4.  Aircraft  Flying  Execution 

5.  Maintenance  Training  Scheduling 

6.  Maintenance  Training  Execution 

7.  Aircraft  Maintenance  Scheduling 

8.  Aircraft  Maintenance  Execution 

9.  Equipment  Maintenance  Scheduling 

10.  Equipment  Maintenance  Execution 

Aircrew  Mission/Training  Scheduling  and  Execution  Process  and 
Flying  Scheduling  and  Execution  Process 
Air  Force  Aircrew  training  is  not  only  paramount  to  the  success  of  the  mission,  it 
is  crucial  to  his  or  her  survival  and  the  survival  of  his  or  her  aircraft  in  a  combat  scenario. 
Aircrews  in  the  CAF  are  among  the  most  comprehensively  trained  in  the  world  as  is 
necessary  to  operate  effectively  in  nearly  any  environment,  in  any  theater,  against  any 
adversary  on  earth.  Aircrews  develop  skills  through  several  years  of  intense  training 
before  ever  entering  the  cockpit  of  a  combat-coded  aircraft.  According  to  AFI  1 1-202 
volume  1 ,  Aircrew  Training, 

The  USAF  Aircrew  Training  Program  (ATP)  ensures  all  aircrew  members  obtain 
and  maintain  the  certification/qualification  and  proficiency  needed  to  effectively 
perform  their  unit’s  mission.  The  objective  of  the  ATP  is  to  develop  and  maintain 
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a  high  state  of  mission  readiness  for  immediate  and  effective  employment  across 
the  range  of  military  operations”  (Department  of  the  Air  Force  Instruction  1 1-202, 
Volume  1,  2010,  p.  3). 

Accumulated  skill  necessary  to  operate  tactical  aircraft  is  developed  from  the 
fundamentals  of  taking  off  and  maintaining  straight  and  level  flight  and  culminates  in 
combat  mission  ready  status  where  he  or  she  is  proficient  in  basic  combat  maneuvers. 
After  fundamentals  for  basic  aircrew  duties  in  the  assigned  aircraft  are  mastered  during 
Initial  Qualification  Training,  the  aircrew  continues  to  hone  skills  in  advanced  aerial 
combat  tactics  during  Continuation  Training  (CT)  and  Upgrade  Training  (UGT).  The  CT 
program  provides  aircrew  members  with  the  volume,  frequency,  and  mix  of  training 
necessary  to  maintain  proficiency  in  the  assigned  certification/qualification  level. 
Aircrews  in  a  combat-coded  unit  may  be  trained  to  the  proficiency  levels  of  Combat 
Mission  Ready  (CMR)  or  the  Basic  Mission  Capable  (BMC).  Aircrew  who  maintain 
qualification  and  proficiency  in  the  command  or  unit  combat  mission  are  considered 
CMR,  while  aircrews  who  are  qualified  in  some  aspect  of  the  unit  mission,  but  do  not 
maintain  CMR  status  are  considered  BMC. 

The  Ready  Aircrew  Program  (RAP)  is  a  training  program  developed  by  local 
commanders  to  align  their  units’  CT  with  the  skills  and  qualifications  required  to  meet 
their  units’  Designed  Operations  Capability  (DOC)  statement  primary  and  secondary 
mission  sets.  The  RAP  Tasking  Message,  sent  to  flying  wings  annually  by  their 
MAJCOM,  “defines  the  minimum  required  mix  of  annual  sorties,  simulator  missions  and 
training  events  aircrew  must  accomplish  to  sustain  combat  mission  readiness”  (HQ 
ACC/A3T,  2013).  RAP  lists  training  and  proficiency  requirements  for  each  level 
according  to  the  crew's  position  in  the  unit  and  qualifications  and  experience  with  the 
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weapons  system.  For  example,  a  typical  inexperienced  Air  Force  pilot  requires  nine 
sorties  in  a  single  month  just  to  stay  proficient  in  take-off  and  landing.  Beyond  the 
fundamental  take-off  and  landing  is  a  myriad  of  other  proficiencies  that  must  be 
maintained  such  as  night  flying,  aerial  refueling,  air-to-ground,  air-to-air,  or  firing  guns 
for  close  air  support.  Flying  squadrons  first  develop  long-tenn  annual  and  quarterly  RAP 
training  plans,  and  continually  refresh  and  refine  these  plans  into  more  short  tenn 
monthly  and  weekly  training  schedules.  This  process  will  be  referred  to  as  the  Aircrew 
Mission/Training  Scheduling  process.  Maintenance  and  Operations  leaders  then  begin 
the  Flying  Scheduling  Process  by  collaborating  to  develop  each  of  these  plans  into  a 
Flying  Schedule  by  communicating  requirements,  understanding  and  considering 
limitations,  and  establishing  a  final,  signed  agreement  between  both  agencies.  The 
Aircrew  Mission/Training  Execution  Process  begins  with  the  development  of  the  mission 
plan.  A  pilot  or  Aircrew  can  spend  days  planning  for  an  upcoming  mission  culminating 
in  the  mission  pre-brief  which  occurs  immediately  before  entering  the  daily  flying  cycle 
as  outlined  in  Figure  3. 
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Figure  3.  Aircrew  Mission/Training  Scheduling  and  Execution  Processes 


Scheduling  aircrew  training  and  mission  planning  are  arduous  processes  for 
operators.  A  single  deviation  during  the  short  term  planning  process  can  disrupt  multiple 
aircrew  training  events  and  cause  significant  scrap  and  schedule  rework  for  both 
operators  and  maintenance.  Maintenance  processes,  specifically  the  delivery  of  mission- 
capable  airplanes  on-time  for  scheduled  events,  are  clearly  the  most  critical  component  to 
the  execution  of  the  schedule.  The  short  term  scheduling  process  is  very  closely  tied  to 
success  of  the  Maintenance  Scheduling  and  Execution  Processes,  which  will  be  described 
in  detail.  Maintenance  failure  to  meet  the  schedule  for  execution  of  planned  missions  can 
turn  a  well  thought  out  plan  into  a  toppling  house  of  cards. 

The  Flying  Execution  Process  begins  with  the  aircrew’s  arrival  at  the  aircraft,  and 
is  another  process  where  aircraft  maintenance  processes  intersect  with  the  flying  process. 
These  intersections  of  processes  continues  to  the  aircrew’s  walk- around  inspection, 
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through  engine  start-up  and  launch  procedures  and  finally  ends  when  the  aircraft  is 
marshaled  out  of  the  end-of-runway  (EOR)  inspection  area. 

Once  an  aircraft  takes-off  and  a  sortie  begins,  the  Aircrew  Mission/Training 
Execution  Process  continues— the  Aircrew  is  required  to  accomplish  a  certain  number  of 
flight  events  for  the  sortie  to  be  considered  effective.  A  non-effective  sortie  is  one  in 
which  no  RAP  mission/sortie  can  be  logged,  and  can  be  caused  by  factors  such  as  aircraft 
system  failures,  air  aborts,  range  weather,  or  in  the  case  of  an  UGT  sortie  the  pilot’s 
failure  to  progress.  Any  one  of  these  factors  causing  a  sortie  to  be  non-effective  is  costly 
as  it  drives  the  need  for  an  additional  sortie  to  be  scheduled  to  complete  the  desired 
training. 

Aircraft  Maintenance  Scheduling  and  Execution  Process 
The  return  on  the  investment  of  training  professional  maintainers  is  realized  when 
the  skills  gained  are  applied  in  the  Aircraft  Maintenance  Scheduling  and  Execution 
processes.  Most  combat-coded  squadrons  have  a  fleet  of  between  18  and  25  aircraft 
assigned  and  ensuring  the  long-term  health  of  the  fleet  demands  the  continuous  attention 
of  maintenance  personnel. 

Aircraft  and  equipment  readiness  is  the  maintenance  mission.  The  maintenance 
function  ensures  assigned  aircraft  and  equipment  are  safe,  serviceable,  and 
properly  configured  to  meet  mission  needs.  Maintenance  actions  include,  but  are 
not  limited  to,  inspection,  repair,  overhaul,  modification,  preservation, 
refurbishment,  troubleshooting,  testing,  and  analyzing  condition  and  performance 
(Department  of  the  Air  Force  Instruction  21-101,  2010,  p.  14)  . 

Each  fleet  of  aircraft  requires  preventative  and  unscheduled  maintenance  actions  in  order 

to  be  safe  for  flight  and  capable  of  performing  any  mission  for  which  the  aircraft  is 

designed.  Scheduled  maintenance  actions  are  perfonned  based  on  prescribed  intervals 
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(hourly-based,  sortie-based,  or  calendar-based)  must  be  accomplished  without  interfering 
with  the  primary  flying  mission.  The  scheduling  of  preventative  maintenance  events 
requires  a  detailed  plan  for  proper  execution  and  to  ensure  an  adequate  number  of  aircraft 
are  available  for  execution  of  the  flying  schedule.  Unscheduled  maintenance  occurs 
either  as  a  result  of  pilot-reported  discrepancies  after  flight,  or  as  ground-found 
discrepancies  discovered  during  inspections  or  scheduled  maintenance.  Both 
preventative  and  unscheduled  maintenance  can  be  grounding  write-ups,  or  flyable  write¬ 
ups  (Department  of  the  Air  Force  TO  00-20-1,  2010).  The  severity  of  the  write-ups  not 
only  effects  airworthiness,  but  also  detennines  the  status  of  the  aircraft  as  compared  to 
the  Mission  Essential  Subsystem  List  published  by  lead  commands.  The  focus  of 
professional  maintenance  leaders  and  managers  is  one  of  knowing  exactly  what  the  status 
of  each  aircraft  is,  assessing  and  setting  priority,  and  allocating  the  proper  resources 
(manpower  and  equipment)  to  restoring  each  aircraft  to  fully  mission  capable  status  in  the 
shortest  amount  of  time  possible.  Figure  4  depicts  the  Aircraft  Maintenance  and 
Scheduling  Process. 
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Figure  4:  Aircraft  Maintenance  Scheduling  and  Execution  Process 


Equipment  Maintenance  Scheduling  and  Execution  Process 
The  success  of  the  flying  wing’s  critical  processes  of  maintaining  aircraft  and 
training  combat-ready  pilots  relies  on  the  availability  of  equipment.  The  combat-ready 
equipment  process  is  necessary  to  ensure  the  correct  assortment  of  equipment  is  available 
to  support  all  scheduled  and  unscheduled  maintenance  events.  Often,  the  equipment  is  so 
complex  or  so  costly  that  MAJCOM  level  management  is  required.  Most  every 
suborganization  within  the  maintenance  group  has  a  hand  in  Equipment  Maintenance,  but 
the  most  visible  Equipment  Maintenance  processes  include: 

1 .  Aerospace  Ground  Equipment  (AGE) 
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2.  Munitions  Material  Handling  Equipment  (MMHE) 

3.  Alternate  Mission  Equipment  (AME) 

4.  Test,  Measurement  and  Diagnostic  Equipment  (TMDE) 

5.  General  tools  and  toolkits 

With  such  a  diverse  and  extensive  list  of  equipment  to  manage  in  a  typical 
Maintenance  Group,  one  could  correctly  ascertain  that  the  requirements  to  keep 
equipment  combat-ready  are  quite  complex.  Each  different  type  of  equipment  has  its 
own  requirements  for  scheduled  maintenance  and  inspections  to  ensure  it  is  combat- 
ready.  Quite  simply,  however,  the  Equipment  Maintenance  Scheduling  and  Execution 
Process  (Figure  5)  looks  similar  to  the  Aircraft  Maintenance  Scheduling  and  Execution 
Process. 
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Figure  5:  Equipment  Maintenance  Scheduling  and  Execution 
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Maintenance  Training  Scheduling  and  Execution  Process 


The  success  of  any  flying  unit  depends  on  the  availability  of  the  right  number  of 
maintainers  with  the  right  skill  sets  and  experience  to  meet  mission  requirements.  Skills 
and  experience  are  a  result  of  training,  and  just  as  in  pilot  training,  the  cumulative 
knowledge  that  our  most  valuable  and  technically  advanced  maintainers  require  comes  at 
significant  expense  over  time.  Without  trained  and  skillful  maintainers  to  generate 
aircraft,  airpower  is  unsustainable,  pilot  training  is  impossible,  and  mission  objectives  are 
unachievable.  AFI  21-101  states: 

Maintenance  training  is  an  essential  element  of  improving  and  sustaining  unit 
capability;  it  must  receive  priority  treatment  by  SQ/CC  and  MOO/MX  SUPT. 
When  balancing  resources  (e.g.,  aircraft,  support  equipment,  facilities,  tools, 
funding,  personnel),  maintenance  training  carries  an  equal  priority  with  the 
operational  training  mission.  Accomplish  maintenance  training  away  from  the 
production/test  environment  (whenever  possible)  to  eliminate/minimize 
distractions  (Department  of  the  Air  Force  Instruction  21-101,  2010,  p.  128) 


Undoubtedly,  the  investment  the  Maintenance  Group  makes  to  develop  combat- 
ready  maintainers  is  worthy  and  comes  with  another  set  of  processes  that  ultimately  add 
value  and  affect  customer  satisfaction.  According  to  AFP  36-2241,  Professional 
Development  Guide,  the  strategy  of  the  Air  Force’s  Education  and  Training  (E&T) 
program  is,  “Develop,  manage,  and  execute  realistic  and  flexible  training  programs  to 
produce  a  highly  skilled,  motivated  force  capable  of  carrying  out  all  tasks  and  functions 
in  support  of  the  Air  Force  mission.  These  programs  should  provide  the  foundation  for 
Air  Force  readiness”  (Department  of  the  Air  Force  Pamphlet  36-2241,  2013,  p.  275) 

The  components  of  the  Maintenance  Group's  Maintenance  Training  Scheduling 
and  Execution  include  on-the-job  training  (OJT),  Upgrade  Training  (UGT),  and 
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Certification  Training.  The  OJT  program  includes  job  knowledge,  job  proficiency  and 
job  experience.  For  maintainers,  the  job  knowledge  component  is  satisfied  by 
successfully  completing  a  career  development  course  (CDC)  that  provides  career 
knowledge,  general  task,  and  deployment/unit  type  code  (UTC)  task  knowledge.  The  Job 
Proficiency  component  is  the  initial  training  an  Airman  receives  at  his  first  work  center 
and  is  achieved  through  hands-on  training  on  tasks  in  the  work  center  (work  center 
requirements).  The  job  experience  component  is  gained  during  and  after  UGT.  UGT  is 
how  an  Airman  progresses  through  the  skill  levels  (3-,  7-,  and  9-skill  level)  and  is 
considered  the  most  vital  piece  to  an  Ainnan’s  total  training  program  (Department  of  the 
Air  Force  Instruction  36-2201,  2010). 

To  achieve  his  3 -skill  level  and  become  an  Apprentice,  an  Airman  must  complete 
an  initial  skills  course  (technical  school).  To  be  a  5-skill  level,  or  Journeyman,  an 
Airman  must  complete  the  CDC  for  his  Air  Force  Specialty  Code  (AFSC)  and  the 
mandatory  core  tasks  outlined  in  the  Career  Field  Education  and  Training  Program 
(CFETP).  Additionally,  the  5-skill  level  requires  a  minimum  of  12  months  of  UGT  and 
meet  the  mandatory  requirements  listed  in  the  Air  Force  Enlisted  Classification  Directory 
(AFECD),  be  recommended  by  his  supervisor  and  approved  by  his  commander 
(Department  of  the  Air  Force  Instruction  36-2101,  2013). 

An  Ainnan  becomes  a  Craftsman  when  he  is  awarded  his  7-skill  level.  To 
achieve  his  7-skill  level  the  Airman  must  complete  a  second  CDC  (7-level  craftsman 
course),  meet  the  mandatory  requirements  listed  in  the  AFECD,  complete  an  additional 
12  months  of  UGT,  be  recommended  by  his  supervisor  and  approved  by  the  commander 
(Department  of  the  Air  Force  Instruction  36-2101,  2013). 
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The  top  three  percent  of  the  enlisted  force  achieve  the  9-skill  level,  or 
Superintendent.  To  be  awarded  the  9-skill  level  the  member  must  be  a  Senior  Master 
Sergeant  or  Chief  Master  Sergeant,  be  recommended  by  his  supervisor  and  approved  by 
the  commander  (Department  of  the  Air  Force  Instruction  36-2101,  2013). 

Developing  a  skillful  maintainer  takes  a  comprehensive  plan  and  requires  a 
tremendous  amount  of  time  and  effort.  In  addition  to  the  skill  level  progression  outlined 
above,  a  maintainer  attends  courses  at  Field  Training  Detachments  (FTD)  throughout  the 
Air  Force  to  gain  fonnal  training  on  specific  systems  such  as  hydraulics,  engines,  or 
advanced  avionics.  Furthering  the  training  regimen  is  recurring  training,  computer  based 
training  and  Professional  Military  Education. 

The  maintainer’ s  bottom-line  mission  is  to  deliver  safe  and  reliable  airplanes  to 
keep  the  war-fighter  in  the  cockpit  as  safe  as  possible  in  an  inherently  dangerous 
environment,  and  the  Maintenance  Training  Scheduling  and  Execution  Processes  (Figure 
6)  are  vital  in  ensuring  that  mission  succeeds. 
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Figure  6:  Maintenance  Training  Scheduling  and  Execution  Process 


Monitoring  and  improving  critical  processes  ultimately  leads  to  achieving 
objectives  and  customer  satisfaction.  Responding  to  the  necessity  of  improving  processes 
takes  careful  consideration  if  resources  are  to  be  used  effectively  and  process  operations 
are  to  be  optimized  to  the  benefit  of  the  customer.  Meaningful  thought  and  action  to 
improve  not  only  add  to  customer  satisfaction,  but  also  ensure  that  processes  stay  ahead 
of  increasing  challenges  and  the  declining  curve  of  resources. 
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Financial  Perspective 

The  Financial  Perspective  of  the  balanced  scorecard  contains  the  big-picture 

metrics  that  gives  the  manager  a  holistic  view  of  whether  the  execution  of  stated  strategy 

is  leading  to  acceptable  end  results.  The  “how”  of  how  an  organization  arrived  at  these 

big  picture  metrics  is  detailed  through  measures  chosen  in  the  other  perspectives. 

We  could  focus  all  of  our  energy  and  capabilities  on  improving  customer 
satisfaction,  quality,  on-time  delivery,  or  any  number  of  things,  but  without  an 
indication  of  their  effect  on  the  organization’s  financial  returns  they  are  of  limited 
value.  Classic  lagging  indicators  are  normally  encountered  in  the  financial 
perspective  (Niven,  2002,  p.  17). 

In  the  corporate  world,  the  shareholders’  perspective  is  of  paramount  importance 
to  business  leaders.  Shareholders  are  driven  by  and  demand  a  return  on  their  investment 
dollars.  For  the  Air  Force,  the  shareholders  are  the  Combatant  Commanders  who  are 
responsible  for  the  prosecution  of  contingency  operations  in  their  area  of  responsibility. 

It  is  vitally  important  for  Maintenance  Group  leaders  to  consider,  “How  do  the 
shareholders  view  us?”  The  Aircraft  Maintenance  community  must  develop  and 
maintain  perfonnance  measures  that  accurately  convey  to  the  Combatant  Commander  the 
bottom-line  readiness  of  the  units  that  are  preparing  to  support  them.  Air  Force 
Personnel  Directive  21-1,  Maintenance,  defines  readiness  as  “The  ability  of  US  military 
forces  to  fight  and  meet  the  demands  of  the  national  military  strategy.  Unit  readiness  is 
the  ability  to  provide  capabilities  required  by  the  combatant  commanders  to  execute  their 
assigned  missions”  (Department  of  the  Air  Force  Policy  Directive  21-1,  2003,  p.  6). 

From  the  Maintenance  Group  standpoint,  one  must  consider  the  critical  assets  under  the 
group’s  control— aircraft,  maintenance  personnel,  and  equipment— and  how  the 
culmination  of  the  previously  described  processes  affects  their  bottom-line  readiness. 
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Learning  and  Growth  Perspective 

Once  you  identify  measures  and  related  initiatives  in  your  Customer  and  Internal 
Process  perspectives,  you  can  be  certain  of  discovering  some  gaps  between  your 
current  organizational  infrastructure  of  employee  skills  and  information  systems, 
and  the  level  necessary  to  achieve  your  results.  The  measures  you  design  in  this 
perspective  will  help  you  close  that  gap  and  ensure  sustainable  performance  for 
the  future.  (Niven,  2002,  p.  16). 

The  Learning  and  Growth  Perspective  applies  to  the  less  tangible  internal 
elements  that  sustain  value  added  processes.  Elements  typically  found  in  the  Learning 
and  Growth  Perspective  are  “enablers”  of  all  the  other  perspectives.  One  might  consider 
technology,  training  or  communications  in  the  Learning  and  Growth  Perspective,  but  for 
the  purpose  of  clarity,  the  researcher  chose  the  maintenance  workforce  because  the 
maintainers  are  the  foremost  and  most  important  enablers  of  all  other  processes  and 
perspectives.  For  the  Maintenance  Group,  the  number  one  internal  element  that  sustains 
the  primary  value  added  processes  is  trained  and  skilled  personnel.  From  the  perspective 
of  the  Maintenance  Group's  employees,  the  maintainers,  a  more  fitting  description  of  the 
"gaps"  between  infrastructure  of  skills  and  the  level  necessary  to  achieve  results  would  be 
the  workforce  quality.  Workforce  quality  could  take  into  account  skill  level  and 
experience  level,  quality  of  maintenance,  safety  of  maintenance,  maintenance  discipline, 
and  retention  rates. 

Maintenance  Group's  Balanced  Scorecard 

While  Kaplan  and  Norton  formulated  the  four  perspectives  outlined  in  Figure  2, 
they  also,  “recognize  these  four  perspectives  should  be  considered  a  template,  not  a 
straight  jacket.  These  perspectives  are  intended  to  portray  the  essential  elements  that  can 
lead  to  success  in  a  typical  organization”  (Kaplan  &  Norton,  1996).  In  other  words,  the 
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creator  of  the  balanced  scorecard  for  any  organization  must  use  some  latitude  to  tailor  the 
framework  to  best  fit  organizational  requirements/structure  and  the  needs  of  leaders  and 
managers  that  will  use  the  balanced  scorecard  to  develop  strategy. 

To  meet  the  unique  requirements  of  a  typical  CAF  Maintenance  Group,  the 
researcher  made  adjustments  to  Kaplan  and  Norton's  balanced  scorecard  to  develop  the 
Proposed  Maintenance  Group  Balanced  Scorecard  (Figure  7).  The  first  perspective 
remains  as  the  Customer  perspective.  Since  a  Maintenance  Group’s  value  is  not 
measured  in  financial  terms,  the  second  perspective  was  renamed  “Readiness”  which 
better  reflects  what  is  expected  by  the  customer  and  stakeholders  (the  combatant 
commander).  The  measures  that  fall  under  “Readiness”— aircraft,  maintenance  personnel, 
and  equipment— tell  us  whether  our  strategy  execution,  which  is  detailed  through 
measures  chosen  in  the  other  perspectives,  is  leading  to  improved  bottom-line  results. 

The  third  perspective  remains  as  the  Processes  Perspective  as  suggested  by  Kaplan  and 
Norton,  while  the  last  perspective  has  been  retitled  “Workforce  Quality”  Perspective. 
Figure  7  depicts  the  Proposed  Maintenance  Group  Balanced  Scorecard  Perspectives  and 
the  Perspective  Categories  that  fall  under  them.  This  proposed  Balanced  Scorecard 
framework  was  used  as  the  model  to  be  evaluated  as  described  in  Chapter  III, 
Methodology. 
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Figure  7:  Proposed  Maintenance  Group  Balanced  Scorecard  Perspectives  and  Perspective 

Categories 
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III.  Methodology 


Overview 

The  purpose  of  this  chapter  is  to  outline  the  methods  used  in  this  study.  First,  a 
background  for  survey  method  of  research  and  why  they  were  chosen  will  be  given, 
followed  by  a  detailed  description  of  the  survey  tool  that  was  administered.  The 
researcher  will  discuss  measures  of  reliability  and  error  addressed  in  the  study,  then 
address  data  preparation  and  the  data  analysis  methodology,  including  statistical  analysis, 
content  analysis,  and  correlation  analysis. 

Method 

The  researcher  chose  to  utilize  a  survey  as  the  method  of  data  collection. 

“Surveys  are  systems  for  collecting  infonnation  from  people  to  describe,  compare,  and 
predict  attitudes,  opinions,  values  and  behavior  based  on  what  people  say  or  see  and  what 
is  contained  in  records  about  them  and  their  activities”  (Fink,  2003).  The  researcher 
chose  to  apply  this  research  method  for  two  reasons: 

1)  “Provides  standardized  measurement  that  is  consistent  across  all  respondents 
and  ensures  that  comparable  information  is  obtained  about  everyone  who  is 
described.”  (Fowler,  2014) 

2)  Probability  sampling  enables  one  to  have  confidence  that  the  sample  is  not  a 
biased  one  and  to  estimate  how  precise  the  data  are  likely  to  be.  Data  from  a 
properly  chosen  sample  are  a  great  improvement  over  data  from  a  sample  who 
attend  meetings,  speak  loudest,  write  letters,  or  happen  to  be  convenient  to 
poll.  (Fowler,  2014) 
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Survey  Formulation  Methodology 

The  researcher  used  the  results  of  the  literature  review  to  formulate  a  survey  to  be 
sent  to  aircraft  maintenance  officers  with  CAF  maintenance  experience.  The  purpose  of 
this  survey  was  to  query  the  maintenance  experts  who  actually  use  the  sanctioned  metrics 
in  the  management  of  their  organizations  to: 

1 .  Explore  the  utility  of  the  balanced  scorecard  framework  for  use  in  a  Maintenance 
Group 

2.  Explore  the  optimal  frequency  of  metrics  analysis  at  each  maintenance  management 
level 

3.  Evaluate  the  "goodness"  of  individual  maintenance  metrics 

The  survey  was  a  cross-sectional  design  that  gathered  descriptive  data  at  one  fixed 
point  in  time,  and  asked  38  questions.  These  questions  included  demographic  questions, 
closed-ended  questions,  multiple  and  single-response  questions,  Likert-scale  questions, 
and  Multiple-rating  matrices,  dynamic  probing  as  well  as  open-ended  questions. 
Demographics 

The  first  section  of  the  survey  asked  for  the  survey  respondent’s  rank,  type  of 
experience  and  level  of  experience  to  establish  demographics  of  the  respondents.  This 
demographic  data  was  used  as  background  information  and  to  differentiate,  analyze, 
trend  and  map  data  survey  responses  in  different  ways. 

Strategic  Objective 

The  purpose  of  the  second  section  of  the  survey  was  to  determine  the  over¬ 
arching  strategic  objective  of  a  typical  Maintenance  Group. 
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Maintenance  Group  Balanced  Scorecard  Perspectives 

The  survey  then  began  to  explore  the  structure  of  the  Maintenance  Group 
Balanced  Scorecard. 

Customer  Perspective 

To  establish  the  structure  for  the  Customer  Perspective,  the  survey  asked 
respondents  whom  the  customers  are  the  Maintenance  Groups  serve  to  achieve  their 
strategic  objective,  in  terms  of  providing  support,  training  or  services.  The  potential 
customers  of  the  Maintenance  Group  outlined  in  Chapter  II— Aircrew,  maintainers, 
aircraft  and  equipment— were  all  offered  as  options,  and  the  respondent  could  choose  as 
many  as  they  thought  were  appropriate. 

Processes  Perspective 

Next,  the  survey  explored  the  Processes  Perspective  of  the  Maintenance  Group 
Balanced  Scorecard.  The  researcher  listed  the  10  processes  outlined  previously,  and 
asked  the  respondents  to  rate  the  relative  importance  of  each  process  in  providing  for  the 
customer  that  the  respondent  had  previously  identified.  The  survey  continued  on  to  ask 
the  respondent  which  of  those  same  processes  they  believe  the  Maintenance  Group  has  an 
impact  on.  At  this  point,  the  researcher  asked  respondents  to  assign  metrics  from  the 
cumulative  sanctioned  metrics  list  in  Appendix  C  to  each  of  the  10  processes.  The 
researcher  also  asked  respondents  if  they  felt  the  set  of  metrics  they  assigned  to  each 
process  were  adequate,  and  if  not,  to  list  and  suggest  metrics  that  may  be  more  adequate 
to  communicate  the  desired  infonnation. 
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Readiness  Perspective 

The  next  section  examined  the  Readiness  Perspective  of  the  Maintenance  Group 
Balanced  Scorecard.  The  purpose  of  this  section  was  to  identify  the  metrics  that  best 
depict  the  readiness  of  Maintenance  Group  entities.  The  survey  outlined  the  three  entities 
of  the  Maintenance  Group— aircraft,  maintenance  personnel,  and  equipment— and  asked 
the  respondents  to  choose  from  the  existing  set  of  metrics  which  best  depict  the  readiness 
of  each  entity.  The  researcher  also  asked  respondents  if  they  felt  the  set  of  metrics 
offered  and  chosen  were  adequate,  and  if  not,  to  suggest  more  adequate  metrics  to  portray 
readiness. 

Workforce  Quality  Perspective 

The  last  portion  of  the  Maintenance  Group  Balanced  Scorecard  part  of  the  survey 
addressed  the  Workforce  Quality  Perspective.  The  purpose  of  this  section  of  the  survey 
was  to  identify  the  metrics  that  best  depict  the  quality  and  skills  of  maintainers.  The 
researcher  outlined  six  indicators  of  Workforce  Quality  for  the  respondents,  and  asked 
the  respondents  to  identify  appropriate  metrics  to  portray  each  of  those  indicators  from 
the  existing  set  of  metrics. 

Balanced  Scorecard  utility  and  frequency 

The  purpose  of  the  next  section  was  to  detennine  the  utility  of  the  Maintenance 
Group  Balanced  Scorecard,  and  the  optimal  frequency  of  analysis  of  the  metrics  in  each 
perspective.  The  survey  asked  respondents  to  assess  the  benefit  of  examining  metrics 
representing  each  perspective  in  the  same  setting,  in  order  for  the  researcher  to  determine 
field  support  or  opposition  of  the  concept.  The  researcher  also  asked  respondents  what 
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they  felt  the  optimal  frequency  of  analysis  would  be  for  the  Maintenance  Group  Balanced 
Scorecard  approach. 

Efficacy  of  individual  metrics 

The  last  section  of  the  survey  asked  respondents  to  evaluate  28  individual 
maintenance  metrics  based  on  their  experience  utilizing  and  analyzing  metrics.  The 
survey  asked  respondents  to  select  every  metric  they  believe  is  not  "good"  for  any  reason, 
and  then  asked  them  to  explain  their  reason  for  each  metric  selected. 

Institutional  and  Air  Force  Approval 

The  researcher  applied  for  Institutional  Review  Board  exemption  from  human 
experimentation  requirements,  since  the  survey  did  not  collect  sensitive  data,  which  could 
reasonably  damage  the  subjects’  financial  standing,  employability,  or  reputation.  The 
demographic  data  collected  also  could  not  map  a  given  response  to  a  specific  subject. 

This  exemption  was  received  through  Air  Force  Institute  of  Technology  review  board  on 
18  December  2014. 

The  researcher  applied  for  a  survey  control  number,  which  is  required  by  Air 
Force  Instruction  38-501,  Air  Force  Survey  Program,  and  received  control  number 
AF14-123AFIT  on  5  February  2014. 

Population  and  Sample 

As  this  research  was  focused  on  Maintenance  Metrics  for  use  in  a  CAF 
Maintenance  Group,  the  population  for  this  research  are  all  members  of  a  CAF 
Maintenance  Group  who  use  maintenance  metrics  to  aid  in  decision  making  and  assess 
the  performance  of  their  organization,  equipment,  aircraft  or  personnel.  This  population 
includes  officers,  enlisted,  contract  or  government  civilians  who  are  involved  in 
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Maintenance  Group  operations,  including  members  of  the  Maintenance  Group, 

Operations  Group,  and  support  agencies. 

The  researcher  elected  to  use  Maintenance  Officers  in  the  grade  of  0-1  through 
0-6  as  the  sample  frame  of  the  population.  These  ranks  typically  serve  at  the 
Maintenance  Group  level,  and  most  directly  apply  the  knowledge  gleaned  from 
maintenance  metrics  analysis.  Maintenance  Officers  in  a  Maintenance  Group  serve  in 
leadership  roles  that  regularly  interact  with  other  decision-makers  that  comprise  the 
population  described  above,  therefore,  have  a  solid  understanding  of  how  metrics  are 
used  by  and  affect  the  entire  population.  Maintenance  Officers  are  charged  with 
understanding  the  big  picture  of  Maintenance  Group  operations,  and  typically  study, 
analyze  and  brief  maintenance  metrics  in  detail  on  a  regular  basis. 

In  order  to  collect  data  from  this  sample,  manpower  data  was  collected  from 
Headquarters  Air  Force  Force  Development  Branch  that  listed  1,406  Air  Force  Aircraft 
Maintenance  Officers  in  the  grade  of  0-1  through  0-6;  however,  the  target  population 
was  actually  a  sub-category  of  maintenance  officers  who  had  experience  in  CAF  flying 
wings. 

Testing  and  Administration 

Pre-Test 

The  survey  was  pre-tested  by  nine  participants  over  six  rounds  of  pre-testing.  A 
pre-test  process  was  conducted  to  ensure  item  specificity,  readability,  representativeness 
and  face  validity.  In  each  round,  nine  individuals  participated  to  complete  the  survey  and 
provide  feedback  about  any  procedural  or  production  problems  (Dillman,  2007).  All  nine 
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participants  were  Maintenance  Officers  with  CAF  experience,  which  made  them  potential 
respondents  as  well,  and  included  two  PhDs,  two  graduate  students,  and  five 
Maintenance  Officers  currently  assigned  to  CAF  flying  wings.  All  nine  that  were  asked  to 
take  the  survey  participated,  for  a  response  rate  of  100%.  Throughout  the  pre-test  process, 
the  survey  was  edited  for  grammar,  content,  and  structure  and  resubmitted  to  participants 
until  the  survey  was  deemed  satisfactory. 

Pilot  Test 

The  survey  was  sent  to  5 1  individuals  during  the  pilot  test,  who  were  all  part  of 
the  sample  frame  described  above.  Out  of  the  5 1  asked  to  complete  the  survey  27 
responded,  for  a  response  rate  of  53%.  All  pilot  test  responses  were  complete  and 
therefore  added  to  the  data  gathered  from  live  survey  implementation. 

Survey  Administration 

The  Maintenance  Metrics  survey,  administered  online  from  10-20  February  2014, 
targeted  Aircraft  Maintenance  Officers  with  experience  in  the  CAF  at  the  flying  wing  and 
Maintenance  Group  level.  The  purpose  of  and  directions  for  the  survey,  authority  for  the 
survey,  as  well  as  a  guarantee  of  confidentiality  and  voluntary  participation  statement 
accompanied  the  invitation  for  survey  completion  sent  by  E-mail  to  each  potential 
participant. 

Survey  Reliability  and  Error 

Reliability 

To  test  the  reliability  of  the  survey,  the  researcher  used  the  test-retest  method, 
which  asks  respondents  to  complete  the  survey  at  two  different  points  in  order  to  measure 
how  stable  the  responses  are.  The  researcher  sent  a  pared-down  version  of  the  final 
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survey,  which  consisted  of  16  questions,  to  eight  survey  respondents  who  had  taken  the 
original  survey  one  week  prior.  The  researcher  then  calculated  the  correlation  coefficient 
between  the  responses  on  the  two  tests  using  the  Gamma  statistic.  The  researcher  chose 
the  Gamma  statistic  to  measure  reliability  because  of  the  ranked  ordinal  nature  of  the 
survey  response  data,  with  a  small  number  of  response  categories.  An  obtained  value  of 
+  1  for  gamma  indicates  the  presence  of  a  perfect  correlation  between  two  variables,  and 
an  obtained  value  of  -1  for  gamma  indicates  the  presence  of  a  perfect  negative  correlation 
(Harding  University,  2014).  The  following  equation  depicts  the  calculation  for  the 
Gamma  statistic: 

Equation  1 :  Gamma  Statistic  fonnula 

C-Na~N, 

Na+N, 

Na  =  Number  of  Agreements 
Ni  =  Number  of  Inversions 

To  detennine  the  significance  of  the  Gamma  statistic,  a  z-score  is  calculated 
based  upon  formula  in  Equation  2.  The  obtained  value  for  the  z-score  will  then  be 
compared  to  the  critical  values  of  z  to  determine  if  the  correlation  is  statistically 
significant.  The  critical  value  for  z  at  the  .05  significance  level  is  +1.96. 

Equation  2:  Gamma  Statistic  z-score  formula 
z  =  G 

Na  =  Number  of  Agreements 
Nj  =  Number  of  Inversions 
N  =  Number  of  Cases 
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The  calculation  for  the  Gamma  statistic  was  based  on  1527  agreements  and  79 
inversions  between  the  variables  (Variable  l=test  responses,  Variable  2=retest 
responses),  and  was  calculated  at  .902,  indicating  a  high  correlation  between  variables,  as 
see  in  Equation  3: 


Equation  3:  Gamma  Statistic  result 


G  = 


N-N,  1527-79  1,448 


Na+N,  1527  +  79  1,606 

Na  =  Number  of  Agreements  =  1527 
N.  =  Number  of  Inversions  =  79 


=  .902  =  G 


The  researcher  calculated  the  z-score  to  be  1.979,  as  shown  in  Equation  4,  which 
fell  within  the  .05  significance  level  critical  value  of  +/-  1.96.  This  means  the  results 
between  the  two  tests  were  found  to  be  statistically  significant,  and  therefore  found  the 
survey  to  be  reliable. 

Equation  4:  Gamma  statistic  z-score  results 
I  1448 

z  =  .902.  / - =  .902  *2.195  =  1.979  =  z 

V  300.45 

Na  =  Number  of  Agreements  =1527 

Nt  =  Number  of  Inversions=79 

N  =  Number  of  Cases=1606 


Sampling  Error 

The  first  type  of  error  the  researcher  addressed  was  sampling  error.  “Sampling 
error  is  the  degree  to  which  the  results  from  the  sample  deviate  from  those  that  would  be 
obtained  from  the  entire  population,  because  of  random  error  in  the  selection  of 
respondent  and  the  corresponding  reduction  in  reliability”  (Alreck  &  Settle,  2004). 
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Given  the  population  size  of  Maintenance  Officers,  the  researcher  determined  that  to 
minimize  sampling  error  and  maximize  reliability  in  the  respondent  results,  she  would 
strive  to  achieve  a  95%  confidence  level,  with  a  confidence  interval  of  plus  or  minus  5%. 
Based  on  these  sampling  error  goals,  the  desired  sample  size,  or  the  number  of 
maintenance  officer  responses  needed,  was  302.  Since  it  was  impossible  to  discern  from 
the  manpower  data  collected  which  of  the  1,406  maintenance  officers  met  the  eligibility 
criteria  of  CAF  experience,  the  survey  was  sent  to  a  sample  frame  of  the  population  (0-1 
to  0-6),  and  used  demographic  data  from  the  respondents  to  determine  who  met  the 
criteria  for  the  subpopulation.  The  researcher  considered  only  the  responses  of  officers 
with  CAF  experience  during  data  analysis. 

Non  Sampling  Error 

Non-Response  Bias 

“Non-response  bias  refers  to  the  mistake  one  expects  to  make  in  estimating  a 
population  characteristic  based  on  a  sample  of  survey  data  in  which,  due  to  non-response, 
certain  types  of  survey  respondents  are  under-represented”  (Berg,  2005).  Since  the 
sample  frame  of  the  survey  was  limited  to  Maintenance  Officers  in  the  grades  of  O-l 
through  0-6,  it  is  possible  that  results  were  affected  by  non-response  bias,  or  bias 
incurred  because  other  members  of  the  population  were  not  surveyed.  Members  of  the 
population  who  were  not  surveyed  include  enlisted,  contract  or  government  personnel, 
and  members  of  the  Operations  Group  and  support  agencies  who  may  utilize  maintenance 
metrics.  Maintenance  Officers  in  grades  higher  than  0-6  also  have  experience  in 
Maintenance  Groups,  but  were  not  surveyed. 
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Response  Bias 

Another  threat  to  validity  of  the  survey  responses  was  respondent  fatigue. 
Respondent  fatigue  occurs  when  people  taking  the  survey  are  affected  by  boredom  or 
lack  of  motivation  to  accurately  answer  the  questions  (Lavrakas,  2008).  The  researcher 
observed  that  only  half  of  respondents  who  began  the  survey  actually  completed  it,  which 
could  have  been  the  result  of  respondent  fatigue.  Incomplete  surveys  were  not 
considered  valid  for  data  analysis  and  those  responses  discarded,  but  there  is  a  possibility 
that  the  remaining  valid  survey  responses  were  not  a  representative  sample  o  the 
population.  There  is  also  a  possibility  that  answers  that  were  considered  valid  were 
actually  affected  by  respondent  fatigue;  however,  these  cases  were  not  as  easy  to  identify. 
When  constructing  the  survey,  the  researcher  made  length  of  the  survey  as  short  as 
possible  without  losing  the  integrity  of  data  being  gathered;  however,  the  survey  was  38 
questions  long  and  could  have  still  introduced  respondent  fatigue  despite  the  researchers 
attempt  to  avoid  it. 

Response  rate 

The  researcher  calculated  the  response  rate  by  dividing  the  number  of  people  who 
submitted  a  completed  survey  (80%  or  more  of  questions  answered)  by  the  number  of 
people  she  contacted  or  attempted  to  contact  to  complete  the  survey.  As  previously 
addressed  in  Chapter  III,  the  researcher  used  the  manpower  data  collected  from 
Headquarters  Air  Force  Development  Branch  to  determine  a  sample  frame  size  of  1,406 
Air  Force  Aircraft  Maintenance  Officers  in  the  grade  of  0-1  through  0-6;  however,  the 
target  population  was  actually  a  sub-category  of  Maintenance  Officers  who  had 
experience  in  CAF  flying  wings.  The  researcher  first  sent  the  survey  to  the  population 
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sample  frame  of  1,406  officers  on  1 1  February  2014  and  followed  up  with  a  subsequent 
invitation  on  18  February  2014.  The  survey  invitation  failed  to  reach  nine  members,  so 
the  population  sample  frame  was  adjusted  to  a  1,397  “solicited  population”  as  depicted  in 
the  “solicited  population”  column  shown  below.  Of  the  1,397  Maintenance  Officers  in 
the  solicited  population,  675  initiated  the  survey,  with  361  completing  it  for  a  54% 
completion  rate,  and  a  26%  total  response  rate.  Of  the  352  completed  surveys,  309  of  the 
officers  indicated  they  had  CAF  experience  for  a  subcategory  response  rate  of  22%. 

These  309  responses  from  the  target  sub-category  were  used  in  the  researcher’s  data 
analysis.  The  completion  rate,  response  rate  and  CAF  experience  rate  by  respondents  is 
depicted  in  Table  1. 


Table  1:  Survey  Response  Rates 


Rank 

Sample 

Frame 

Population 

Solicited 

Population 

Total 

Started 

Survey 

Total 

Completed 

Survey 

Completion 

rate 

Response  Rate 

CAF 

Experience 

%of 

Resp 

2Lt 

118 

118 

43 

23 

53.5% 

19.5% 

11 

9.3% 

lLt 

175 

174 

70 

35 

50.0% 

20.1% 

23 

13.2% 

Capt 

397 

396 

188 

105 

55.9% 

26.5% 

89 

22.5% 

Maj 

306 

306 

167 

81 

48.5% 

26.5% 

78 

25.5% 

LtCol 

291 

286 

144 

83 

57.6% 

29.0% 

75 

26.2% 

Col 

119 

117 

63 

34 

54.0% 

29.1% 

33 

28.2% 

Total 

1406 

1397 

675 

361 

53.5% 

25.8% 

309 

22.1% 

Demographics 

The  first  section  of  the  survey  asked  for  the  survey  respondent’s  rank,  type  of 
experience  and  level  of  experience  to  establish  demographics  of  the  respondents.  This 
demographic  data,  as  shown  in  Table  2,  is  used  as  background  information  and  to 
differentiate,  analyze,  trend  and  map  data  survey  responses  in  different  ways.  The 
demographic  data  reveals  that  the  majority  of  the  survey  respondents  were  in  the  ranks  of 
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0-3  through  0-5.  Almost  50%  of  the  respondents  held  positions  in  the  Maintenance 
Group  (AMU  OIC,  AMXS  Operations,  Squadron  Commander,  Deputy  or  Group 
Commander)  that  actually  have  direct  ownership/authority  over  aircraft  maintenance  and 
aircraft  maintainers.  Over  50%  of  the  respondents  have  great  than  6  years  in  a  CAF 
flying  wing.  Over  60%  have  between  6  and  20  plus  years  analyzing/reporting 
maintenance  metrics  and  75%  had  between  6  and  20  plus  years  experience  in  the  aircraft 
maintenance  arena.  The  demographic  data  lends  credibility  to  the  information  and 
opinions  captured  because  the  largest  proportion  of  respondents  operate  or  have  operated 
in  aircraft  maintenance  for  a  sufficient  amount  of  time  to  be  considered  experts  in 
positions  most  relevant  to  the  aircraft  maintenance  profession. 


Table  2:  Demographics 


1.  What  is  your  rank? 

Answer  Options 

Response  % 

Response  Count 

2Lt  (O-l) 

3.6% 

11 

lLt  (0-2) 

7.5% 

23 

Captain  (0-3) 

28.9% 

89 

Major  (0-4) 

25.0% 

77 

LtCol  (0-5) 

24.4% 

75 

Colonel  (0-6) 

10.7% 

33 

2.  Which  most  closely  describes  your  current  MAJCOM  (or  equivalent)? 

Answer  Options 

Response  % 

Response  Count 

ACC 

24.0% 

74 

AETC 

11.7% 

36 

AFGSC 

5.2% 

16 

AFMC 

13.3% 

41 

AFR 

0.6% 

2 

AFSC 

0.0% 

0 

AFSOC 

3.9% 

12 

AMC 

12.7% 

39 

ANG 

0.3% 

1 

Direct  Reporting  Unit  (DRU) 

1.9% 

6 

DLA 

0.3% 

1 
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Table  3:  Demographics 


Forward  Operating  Agency  (FOA) 

0.0% 

0 

HAF 

3.9% 

12 

PACAF 

9.4% 

29 

USAFCENT 

3.2% 

10 

USAFE 

4.9% 

15 

Other  (please  specify) 

4.5% 

14 

3.  What  is  the  management  level  of  your  current  position 

> 

Answer  Options 

Response  % 

Response  Count 

Flight  Commander 

3.9% 

12 

AMU  OIC 

12.8% 

39 

EM  S/CMS/MX  S/Muns/MO  S  Operations  Officer 

5.3% 

16 

AMXS  Operations  Officer 

8.9% 

27 

Squadron  Commander 

14.5% 

44 

Deputy  or  Group  Commander 

14.8% 

45 

MAJCOM 

7.9% 

24 

Depot 

2.3% 

7 

Other  (please  specify) 

29.6% 

90 

4.  How  many  years  of  experience  do  you  have  in  Aircraft  Maintenance? 

Answer  Options 

Response  % 

Response  Count 

Less  than  1  year 

1.0% 

3 

1-3  years 

8.8% 

27 

4-6  years 

11.4% 

35 

6-10  years 

15.9% 

49 

10-15  years 

15.6% 

48 

15-20  years 

19.8% 

61 

20+  years 

27.6% 

85 

5.  How  many  years  of  experience  do  you  have  in  a  Combat  Air  Forces  (CAF)  flying  wing? 

Answer  Options 

Response  % 

Response  Count 

None 

0.0% 

0 

Less  than  1  year 

4.2% 

13 

1-3  years 

25.3% 

78 

4-6  years 

19.5% 

60 

6-10  years 

22.7% 

70 

10-15  years 

15.9% 

49 

15-20  years 

6.8% 

21 

20+  years 

5.5% 

17 

6.  What  levels  of  maintenance  have  you  managed?  Please  select  all  that  apply. 

Answer  Options 

Response  % 

Response  Count 

Flight  Commander 

96.1% 

296 

AMU  OIC 

89.9% 

277 
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Table  4:  Demographics 


EM  S/CMS/MX  S/Muns/MO  S  Operations  Officer 

62.0% 

191 

AMXS  Operations  Officer/Squadron  Maintenance  Officer  (SMO) 

59.4% 

183 

Squadron  Commander 

48.7% 

150 

Deputy  or  Group  Commander 

23.1% 

71 

MAJCOM 

26.0% 

80 

Depot 

24.0% 

74 

Other  (please  specify) 

14.3% 

44 

7.  How  much  experience  do  you  have  analyzing  or  reporting  maintenance  metrics? 

Answer  Options 

Response  % 

Response  Count 

Less  than  1  year 

3.6% 

11 

1-3  years 

12.4% 

38 

4-6  years 

19.5% 

60 

6-10  years 

20.8% 

64 

10-15  years 

21.5% 

66 

15-20  years 

13.7% 

42 

20+  years 

8.5% 

26 

Data  Preparation 

Data  Inspection 

The  researcher  inspected  the  data  for  errors  that  could  have  occurred  during  data 
entry  or  errors  resulting  from  respondents’  inconsistent  answers.  The  most  conspicuous 
errors  the  researcher  searched  for  were  incomplete  survey  responses,  which  were  not 
considered  in  the  analysis. 

Closed-ended  survey  responses 

The  researcher  imported  closed-ended  survey  responses  into  Excel  to  perfonn 
descriptive  statistical  analyses,  response  rates,  and  frequencies.  Close-ended  survey 
responses  were  coded  using  pre-weighted  Likert  scales  (l=Strongly  Disagree  to 
5=Strongly  Agree),  or  used  binary  yes/no  responses. 
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Open-ended  Survey  Responses 

The  researcher  used  a  coding  mechanism  to  organize  content  and  reveal  trends  or 
patterns  in  open-ended  survey  responses.  The  coding  mechanism  entailed  categorizing 
responses  and  assigning  a  numerical  code  to  each  category,  then  entering  into  excel  for 
further  statistical  analysis  such  as  frequency  distribution,  central  tendency,  and 
variability. 

Survey  Analysis  Methodology 

Statistical  Analysis 

The  researcher  used  a  three-part  process  to  analyze  the  metrics  that  respondents 
assigned  to  perspective  categories, 

First,  the  researcher  applied  an  initial  filter  of  metrics  to  consider  under  each 
Balanced  Scorecard.  The  mean  and  standard  deviation  of  metric  selections  under  each 
perspective  category  were  calculated  to  perfonn  this  filter.  Chebyshev’s  Rule  in  statistics 
states  that  no  useful  information  is  provided  on  the  fraction  of  measurements  that  fall 
within  one  standard  deviation  of  the  mean  (McClave,  et  ah,  2011).  Therefore,  any 
metrics  that  garnered  enough  selections  to  fall  above  one  standard  deviation  from  the 
mean  number  of  responses  were  considered  for  inclusion  in  the  Maintenance  Group 
Balanced  Scorecard,  while  any  metrics  that  fell  below  the  one  standard  deviation 
standard  were  excluded. 

Next,  the  researcher  detennined  that  a  50%  selection  rate  for  a  metric  assignment 
to  a  perspective  category  was  the  minimum  threshold  for  consideration— if  a  metric  had 
less  than  50%  of  respondents  who  assigned  it  to  the  process  under  review,  it  was 
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eliminated  from  consideration  for  inclusion  to  the  scorecard  (Warr,  2014).  This  filter 
eliminated  most  illogical  and  unrelated  metric  and  perspective  category  combinations, 
such  as  Abort  Rate  assignment  to  the  Equipment  Maintenance  Scheduling  Process.  This 
filter  also  prevented  metrics  from  being  selected  that  met  the  one  standard  deviation  from 
the  mean  criteria  described  above,  but  were  part  of  a  data  set  with  too  low  an  average  and 
standard  deviation  to  be  meaningful  for  this  study. 

Finally,  the  researcher  analyzed  the  data  that  indicated  the  respondents’  feelings 
on  whether  the  available  metrics  adequately  measured  the  efficiency  and  effectiveness  of 
the  perspective  categories  under  consideration.  Four  data  pennutations  were  possible 
when  comparing  respondents’  feelings  regarding  metric  adequacy  data,  and  the  actual 
metrics  assignment  data: 

1 .  Respondents  felt  the  metrics  presented  were  adequate,  and  at  least  one  metric  fell 
above  one  standard  deviation  from  the  mean 

2.  Respondents  felt  measures  were  adequate,  but  no  measures  fell  above  one  standard 
deviation  from  the  mean 

3.  Respondents  felt  measures  were  inadequate,  but  at  least  one  measure  fell  above  one 
standard  deviation  from  the  mean 

4.  Respondents  felt  measures  were  inadequate,  and  no  measures  fell  above  one  standard 
deviation  from  the  mean. 

If  the  results  fell  into  any  of  these  categories  with  the  exception  of  first,  the 
researcher  would  perform  additional  Content  Analysis  using  respondents’  comments 
relating  to  these  metrics  and  processes. 
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Content  Analysis  and  Qualitative  Validity 

The  researcher  opted  to  use  an  open-ended  question  to  let  respondents  state,  in 
their  own  words,  what  problems  they  had  with  individual  metrics,  providing  them 
“freedom  in  framing  the  answers”  (Weisberg  &  Bowen,  1977,  p.  49).  Providing  this  type 
of  freedom  required  coding  to  organize  content  and  properly  analyze  and  trend  the 
responses.  The  researcher  used  manifest  coding  to  evaluate  the  substance  of  the 
respondent’s  answer  to  a  question  when  responses  did  not  meet  one  of  both  criteria  listed 
above.  To  develop  the  manifest  codes  for  the  open-ended  questions,  the  researcher  used 
a  mix  of  the  theoretical  approach  and  the  contextual  approach. 

The  theoretical  method  develops  codes  based  on  expected  answers,  which  the 
researcher  used  in  the  analysis  of  efficacy  of  individual  metrics  (Weisberg  &  Bowen, 
1977).  The  researcher  used  the  Caplice  and  Sheffi  metrics  evaluation  criteria  discussed 
in  Chapter  II  as  the  theoretical  method  to  code  these  open-ended  responses  into  one  of  the 
eight  criterion  categories. 

For  perspective  category  coding,  the  researcher  used  the  contextual  method.  This 
approach  codes  answers  based  on  responses  received,  which  was  of  utility  to  the 
researcher  since  there  were  no  preconceptions  about  the  answers  she  would  receive  to 
open-ended  questions  through  the  majority  of  the  survey.  (Weisberg  &  Bowen,  1977). 
The  researcher  first  sought  to  classify  and  trend  respondents’  issues  with  available 
metrics;  based  on  respondents’  answers,  the  researcher  developed  the  following 
categories: 

1 .  Not  classifiable 

2.  Don't  need  to  track 
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3.  No  metrics  available  but  should  be 


4.  Metrics  available  and  looked  at  MXG  level  but  not  listed  as  choices 

5.  Metrics  Available  from  external  sources  but  not  visible  to  MXG 

6.  Metrics  available  from  internal  sources  but  not  looked  at  MXG  level 

7.  Metrics  available  but  are  inadequate 

The  researcher  then  sought  to  classify  and  trend  proposed  metrics  by  respondents.  The 
classifications  for  proposed  metrics  were  unique  to  the  perspective  category  in  question. 

To  ensure  validity  of  the  coding  process,  the  researcher  first  independently 
analyzed  comments  for  the  perspective  categories  that  required  additional  analysis,  and 
then  utilized  an  external  auditor  to  review  the  qualitative  study  and  provide  an  objective 
assessment  of  the  coding  (Lincoln  &  Guba,  1985).  The  researcher  and  external  auditor 
reviewed  and  discussed  respective  individual  coding  results.  A  comparison  of  the 
individually  coded  items  by  the  researcher  and  the  external  auditor  resulted  in  a 
confirmation  rate  of  92%. 

Correlation  Analysis 

The  researcher  performed  a  correlation  analysis  to  examine  the  relationship 
between  perspective  categories  and  the  selection  of  metrics  used  to  represent  each 
category.  Correlation  analysis  is  to  measure  the  linear  relationship  or  association 
between  defined  variables;  the  resulting  correlation  coefficient  (r  or  rho)  indicates  how 
closely  the  data  fit  a  linear  pattern.  A  positive  correlation  coefficient  indicates  that  an 
increase  in  one  variable  corresponds  to  an  increase  in  the  other  variable,  implying  a 
relationship  between  the  two.  A  negative  correlation  indicates  the  opposite;  when  one 
variable  increases,  the  other  decreases  (Taylor,  1990).  Correlation  coefficient  values  fall 
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between  -1  and  1.  Values  falling  under  0.35  are  generally  considered  to  represent  low  or 
weak  correlations,  and  values  falling  from  0.36  to  0.67  are  considered  modest  or 
moderate  correlations.  Values  falling  between  0.68  and  1.0  can  be  considered  strong  or 
high  correlations  and  values  greater  than  0.90  can  be  considered  very  high  correlations 
(Taylor,  1990). 

A  correlation  coefficient  (r)  was  used  to  examine  the  relationship  between 
perspective  categories  with  regard  to  selection  of  metrics  used  to  represent  each  category. 
A  high  correlation  coefficient  when  comparing  two  perspective  categories  indicates 
strong  agreement  of  same  metrics  selected  to  measure  those  categories,  while  a  low 
correlation  coefficient  indicates  weak  agreement  among  the  choice  of  metrics  selected. 
This  infonnation  can  assist  in  determining  if  perspective  categories  selected  are 
redundant  in  nature  and  should  be  eliminated  from  the  Maintenance  Group  Balanced 
Scorecard,  or  if  available  metrics  should  be  tailored  to  better  measure  the  unique 
purposes  of  each  perspective  category  in  question. 

JMP  10.0  provided  the  correlation  analysis.  The  data  was  normalized  by 
converting  raw  number  of  responses  into  percentages  of  total  respondents  per  perspective 
category,  since  the  total  number  of  respondents  differed  from  category  to  category.  Each 
pair  of  perspective  category  entries  were  placed  in  a  scatterplot,  and  a  best  fit  regression 
line  used  to  indicate  the  degree  of  correlation  between  the  two  perspective  categories 
under  consideration.  For  example,  the  regression  line  of  the  plot  of  perspective 
categories  A  and  B  in  Figure  8  (y  =  -.1397,  x  +  .4302)  shows  two  perspective  categories 
with  a  relatively  low  slope  angle  (m  =  -.13),  or  correlation. 


59 


100.0% 

90.0% 

80.0% 

70.0% 

60.0% 

50.0% 

40.0% 

30.0% 

20.0% 

10.0% 

0.0% 


♦  Metrics  1-28 


y=  -0.1397X  + 0.4302 


3.0%  10.0%  20.0%  30.0%  40.0%  50.0%  60.0% 

Perspective  Category  A,  %  responses 


Figure  8:  Example  Perspective  Category  Regression  Plot 


A  high  correlation  coefficient  between  two  perspective  categories  indicates  a  high 
agreement  of  votes  for  the  metrics  selected  to  measure  those  categories.  A  low  correlation 
coefficient  indicates  a  low  agreement  of  votes  for  the  metrics  selected  to  measure  them. 
As  outlined  in  Figure  7,  the  19  perspective  categories  were: 

1 .  Flying  Scheduling  Process 

2.  Flying  Execution  Process 

3.  Aircraft  Maintenance  scheduling  process 

4.  Aircraft  Maintenance  execution  process 

5.  Equipment  Maintenance  scheduling 

6.  Equipment  Maintenance  execution 

7.  Maintenance  Training  Scheduling 

8.  Maintenance  Training  Execution 

9.  Aircrew  Mission/  Training  scheduling 

10.  Aircrew  Mission/  Training  execution 
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1 1 .  Maintainer  readiness 

12.  Aircraft  readiness 

13.  Equipment  readiness 

14.  Maintenance  quality 

15.  Maintenance  safety 

16.  Maintainer  skill  level 

17.  Maintainer  experience  level 

18.  Maintainer  Discipline 

19.  Maintainer  Retention 

For  the  171  combination  pairs  of  19  perspective  categories,  the  researcher 
detennined  that  any  r-value  that  fell  above  .68  would  be  further  analyzed,  as  values  in 
that  range  are  generally  considered  high  correlations. 

While  it  is  true  that  metrics  are  designed  to  intertwine  and  can  be  used  to 
corroborate  concerns  rising  from  other  metrics  and  therefore  should  be  correlated,  the 
researcher  sought  to  identify  high  correlations  where  they  shouldn’t  logically  exist.  For 
example,  an  extremely  high  correlation— or  nearly  exact  selections  of  metrics  and 
magnitude  of  those  selections— indicates  either  the  right  mix  of  metrics  is  not  available  to 
portray  the  desired  information,  or  the  suggested  perspective  categories  under 
consideration  are  redundant.  This  infonnation  can  substantiate  the  previously  perfonned 
statistical  and  Content  Analysis,  and  assist  in  detennining  if  a  perspective  category 
should  be  eliminated  from  the  Maintenance  Group  Balanced  Scorecard,  or  if  available 
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metrics  should  be  tailored  or  added  to  better  measure  the  unique  purposes  of  each 
perspective  category  in  question. 
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IV.  Results  and  Analysis 


Overview 

In  this  chapter,  the  researcher  will  first  present  the  findings  from  the  survey 
research  relating  to  the  Maintenance  Group  Balanced  Scorecard,  and  the  Perspectives  and 
metrics  contained  therein.  The  researcher  will  present  the  results  of  the  statistical 
analysis,  content  analysis  and  correlation  Analysis,  as  well  findings  on  the  efficacy  of 
individual  metrics  to  refine  the  proposed  Maintenance  Group  Balanced  Scorecard 
presented  in  Chapter  III. 

Statistical  Analysis 

Strategic  Objective 

The  first  survey  question  sought  to  answer  the  researcher’s  first  research  question, 
what  is  the  general  strategic  objective  of  a  Maintenance  Group?  When  asked  what  their 
assessment  of  the  statement,  “The  primary  strategic  objective  of  a  CAF  Maintenance 
Group  is  to  maintain  air  and  space  equipment  in  a  safe,  serviceable  and  ready  condition  to 
facilitate  mission-readiness  of  the  flying  wing”,  97%  of  the  respondents  indicated  that 
they  agreed  or  strongly  agreed  with  this  statement.  The  comments  following  this 
question  established  the  foundation  for  the  Balanced  Scorecard,  and  helped  to  clearly 
define  which  customers  must  be  served  to  achieve  the  strategic  objective,  and  the  critical 
processes  required  to  serve  those  customers.  Over  25%  of  respondents  added  that  while 
the  definition  was  mostly  correct,  CAF  units  do  not  work  with  space  assets,  and  noted  the 
lack  of  mention  of  personnel.  For  this  reason,  the  researcher  refined  the  general  strategic 
objective  of  the  Maintenance  Group  to  read,  “Maintain  aircraft,  equipment  and  personnel 
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in  a  safe,  serviceable  and  ready  condition  to  facilitate  the  mission-readiness  of  the  flying 
wing.” 

Maintenance  Group  Balanced  Scorecard  perspectives 

With  the  strategic  objective  clearly  identified  by  the  respondents,  the  survey  then 
began  to  further  explore  the  structure  of  the  Maintenance  Group  Balanced  Scorecard  that 
includes  the  Customer  Perspective,  Process  Perspective,  Readiness  Perspective  and  the 
Workforce  Quality  Perspective.  As  previously  discussed,  these  perspectives  provide 
specific  answers  to  simple  questions  that  keep  an  organization  on  task  to  excel  at  specific 
processes  to  benefit  a  clearly  defined  customer. 

Customer  Perspective 

To  clarify  and  solidify  the  structure  for  the  Customer  Perspective,  the  survey 
asked  respondents  whom  the  customers  are  the  Maintenance  Groups  serve  to  achieve 
their  strategic  objective,  in  terms  of  providing  support,  training  or  services.  With 
maintainers,  Aircrew,  aircraft  and  equipment  as  their  options,  88%  of  the  respondents 
stated  that  the  Maintenance  Group  should  view  the  Aircrew  as  customers,  and  50%  of  the 
respondents  stated  that  the  Maintenance  Group  should  view  maintainers  as  customers. 
Although  a  little  more  than  half  the  population  considered  maintainers  as  customers,  the 
number  of  positive  responses  did  not  fall  above  one  standard  deviation  from  the  mean 
number  of  responses  for  all  processes,  and  therefore  was  deemed  insignificant.  Figure  9 
shows  the  breakdown  of  responses  on  who  the  customer  is  the  Maintenance  Group 
provides  for.  As  there  was  a  clear  preference  indicated  by  respondents,  and  that 
preference  met  the  filter  and  standard  deviation  criteria,  no  Content  Analysis  was 
required  to  substantiate  these  findings. 
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Figure  9:  MXG’s  Customer(s)  Responses 


Process  Perspective 

The  identification  of  the  Aircrew  as  the  Customer  was  used  as  the  point  of 
reference  when  identifying  the  Maintenance  Group’s  critical  processes  in  providing  for 
customers.  In  the  survey,  the  researcher  listed  the  10  processes  outlined  in  Chapter  II  and 
asked  the  respondents  to  rate  the  relative  importance  of  each  process  in  providing  for  the 
Aircrew  as  the  customer.  Figure  10  depicts  that  nearly  80%  of  respondents  of  the  272 
respondents  felt  that  Aircraft  Flying  Scheduling  and  Execution  were  critical  processes  in 
providing  for  the  Aircrew  as  the  customer.  Aircrew  Mission/Training  Scheduling  and 
Execution  also  fell  above  one  standard  deviation  of  the  average  number  of  responses  for 
all  processes  and  were  included  as  critical  processes  for  serving  Aircrew  as  the  customer 
as  well.  These  processes  will  be  classified  under  the  Customer  Perspective  of  the 
Maintenance  Group  Balanced  Scorecard  as  they  most  critically  affect  the  customer. 
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Figure  10:  Critical  Processes  for  Aircrew  as  Customer 


Aircraft  Maintenance  Scheduling  and  Execution,  Equipment  Maintenance 
Scheduling  and  Execution,  and  Maintenance  Training  Scheduling  and  Execution  will 
therefore  be  considered  Internal  processes;  although  these  processes  do  not  directly 
impact  the  Aircrew  as  the  customer  according  to  respondents,  these  processes  indirectly 
provide  for  the  Customer;  about  60%  of  respondents  classified  these  processes  as 
“important  or  moderately  important”  (Figure  11). 
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Figure  1 1 :  Important  Processes  for  Aircrew  as  Customer 


Since  the  Customer  of  the  Maintenance  Group  (Figure  9),  the  Critical  Processes 
for  that  customer  (Figure  10)  and  Important  Internal  Processes  (Figure  1 1)  have  been 
established,  the  next  step  was  to  assign  metrics  that  best  portray  the  efficiency  and 
effectiveness  of  those  processes. 

The  survey  asked  respondents  to  assign  metrics  (listed  in  Appendix  C)  to  each  of 
the  ten  processes,  selecting  as  many  metrics  for  each  process  that  they  deemed 
appropriate.  Appendix  H  depicts  the  percentage  of  respondents  who  selected  each  metric 
for  each  process. 

The  next  portion  will  describe  the  results  of  this  three-part  analysis  for  each  of  the 
Process  categories  as  described  in  the  Chapter  III.  First,  the  fdtering  process  was  applied 
for  metric  selection,  followed  by  analysis  of  respondents’  perception  of  the  adequacy  of 
the  metrics  set  for  each  process.  Finally,  a  content  analysis  was  performed  when  the 


67 


filtering  and  metric  adequacy  analysis  did  not  paint  a  clear  picture  of  what  metrics  were 
appropriate  to  use. 

Aircrew  Mission/T raining  Scheduling  and  Execution 
Filtering  Process:  In  the  Aircrew  Mission/Training  Scheduling  Process  category,  only 
two  metrics  met  the  50%  selection  rate  and  fell  above  one  standard  deviation  from  the 
mean:  Flying  Scheduling  Effectiveness  and  Average  Sortie  Duration  (Figure  12).  The 
Execution  category  had  four  metrics  that  met  the  inclusion  criteria:  Abort  rate,  Average 
Sortie  Duration,  Deviation  rate,  and  Flying  Scheduling  Effectiveness  (Figure  13). 
Respondents  perceived  these  metrics  the  most  relevant  to  the  Aircrew  Mission/Training 
Scheduling  processes  and  should  be  considered  as  part  of  the  Maintenance  Group’s 
Balanced  Scorecard. 


Mean  =  24.8%  _.  _  .  _  .  ... 

0=13%  f  37  9%  Aircrew  Mission/ Training  Scheduling 


Figure  12:  Metric  Assignment  to  Aircrew  Mission/Training  Scheduling  Process 
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Figure  13:  Metric  Assignment  to  Aircrew  Mission/Training  Execution  Process 


Respondent  analysis  of  adequacy:  Less  than  50%  of  the  respondents  felt  there 
was  adequate  measure  of  the  Aircrew  Mission/Training  Scheduling  and  Execution 
processes.  (Figure  14). 
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Figure  14:  Adequacy  of  Available  Metrics  for  Aircrew  Mission/Training  Scheduling 
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Although  at  least  one  measure  fell  above  one  standard  deviation  from  the  mean 
inclusion  requirement,  respondents  felt  measures  were  inadequate;  therefore,  content 
analysis  was  required  to  reveal  reasons  for  the  perceived  inadequacy. 

Trend  issues  with  available  metrics:  Content  analysis  of  comments  offered  by 
respondents  regarding  issues  they  had  with  metrics  representing  the  Aircrew 
Mission/Training  Scheduling  and  Execution  Processes  overwhelmingly  pointed  to  a  lack 
of  visibility  of  metrics  that  are  studied  by  the  Operations  Group  that  could  be  of 
significant  value  to  the  Maintenance  Group.  Comments  indicate  the  need  for  plainly 
visible  measures  for  progress  toward  specific  goals  for  Operations  that  are  discussed  in 
an  open  forum  with  Maintenance,  so  Maintenance  is  able  to  see  how  their  processes  truly 
impact  Operations  schedules.  Current  metrics  do  not  clearly  articulate  if  there  were 
deviations  to  scheduled  missions,  only  if  there  were  deviations  to  the  scheduled  launch  of 
the  sortie.  Comments  also  suggest  there  is  a  lack  of  metrics  to  understand  if  Operations 
is  being  judicious  with  the  aircraft  provided  to  them,  as  well  as  a  lack  of  metrics  to  be 
able  to  substantiate  the  need  for  such  things  as  adding  sorties  or  making  last  minute 
changes  to  the  schedule.  Decisions  appear  to  be  assertions  based  on  anecdote  and  often 
put  Maintenance  in  a  reactive  posture.  Bottom  line,  the  comments  indicate  a  lack  of 
transparency  and  understanding  in  Operation’s  requirements  and  exactly  how 
Maintenance  impacts  them  (Figure  15). 
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Aircrew  Mission/TrainingScheduling  and  Execution: 
Trended  Issues  with  Available  Metrics 
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Figure  15:  Aircrew  Mission/Training  Scheduling  and  Execution  Process  Metrics  Issues 


Trend  suggested  metrics:  The  majority  of  respondents  pointed  to  an  effectiveness 
metric  as  a  useful  tool  in  knowing  how  the  Maintenance  Group  is  serving  the  customer. 

A  Mission  Effectiveness  Rate  yielded  the  most  suggestions  by  far,  to  help  maintainers  to 
understand  with  more  fidelity  if  Aircrew  accomplished  the  mission  as  planned  and 
scheduled.  Additional  suggestions  further  refined  the  Mission  Effectiveness  Rate;  some 
respondents  suggested  metrics  that  articulate  thrash  in  the  Aircrew  Scheduling  Process. 
For  example,  a  rate  that  shows  deviations  between  scheduled  missions  and  planned 
missions,  a  rate  that  shows  deviations  between  planned  missions  and  briefed  missions, 
and  a  rate  that  shows  deviations  between  briefed  missions  and  flown  missions. 

Deviations  to  any  parts  of  the  Scheduling/Planning/Execution  process  would  be  attributed 
to  the  responsible  agency— similar  to  traditional  Flying  Scheduling  Effectiveness  rate— to 
understand  what  is  causing  thrash  to  the  schedule  and  why  (Figure  16). 
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Figure  16:  Aircrew  Mission/Training  Scheduling  Process  Suggested  Metrics 


Flying  Scheduling  and  Execution 

Filtering  Process:  Six  metrics  met  the  inclusion  criteria  for  the  Flying  Scheduling 
Process.  These  included  Aircraft  Availability,  Flying  Scheduling  Effectiveness, 
Maintenance  Scheduling  Effectiveness,  Phase/Isochronal  Inspection,  Sorties  Scheduled 
and  UTE  Rate  (Figure  17).  Five  metrics  met  the  50%  selection  rate  for  the  Flying 
Execution  Process:  Abort  Rate,  Average  Sortie  Duration,  Break  Rate,  Deviation  Rate 
and  Flying  Scheduling  Effectiveness  (Figure  18).  Respondents  perceived  these  metrics 
as  the  most  relevant  to  the  Aircraft  Flying  Scheduling  and  Execution  Processes  and 
should  be  considered  as  part  of  the  Maintenance  Group’s  Balanced  Scorecard. 
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Figure  17:  Metric  Assignment  to  Flying  Scheduling  Process 


Figure  18:  Metric  Assignment  to  Flying  Execution  Process 


Respondent  analysis  of  adequacy:  The  response  rate  met  both  the  50%  agreement 
rate,  and  exceeded  one  standard  deviation  from  the  mean  responses.  Approximately  78% 
of  survey  respondents  indicated  the  metrics  currently  available  for  the  Flying  Scheduling 
Process  are  adequate  (Figure  19). 
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Figure  19:  Adequacy  of  Available  Metrics  for  Flying  Scheduling  Process 


Respondents  felt  measures  were  adequate,  and  at  least  one  measure  fell  above  one 
standard  deviation  from  the  mean— content  analysis  was  not  required. 

Maintenance  Training  Scheduling  and  Execution 
Filtering  Process:  Three  metrics  for  the  Maintenance  Training  Scheduling 
Process  met  the  inclusion  criteria:  CDC  Pass  Rate,  Training  Rate  and  the  Upgrade 
Training  Rate  (Figure  20).  Three  metrics  for  the  Maintenance  Training  Execution 
Process  met  the  inclusion  criteria:  CDC  pass  rate,  Training  Rate  and  the  Upgrade 
Training  Rate  (Figure  21).  Respondents  perceived  these  metrics  the  most  relevant  to  the 
Maintenance  Training  Scheduling  and  Execution  Processes  and  should  be  considered  as 
part  of  the  Maintenance  Group’s  Balanced  Scorecard. 
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Figure  20:  Metric  Assignment  to  Maintenance  Training  Scheduling  Process 


Figure  2 1 :  Metric  Assignment  to  Maintenance  Training  Execution  Process 


Respondent  analysis  of  adequacy:  The  response  rate  met  both  the  50%  agreement 
rate,  and  exceeded  one  standard  deviation  from  the  mean  responses  (Figure  22). 
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Figure  22:  Adequacy  of  Available  Metrics  for  Maintenance  Training  Scheduling  Process 


Respondents  felt  measures  were  adequate,  and  at  least  one  measure  fell  above  one 
standard  deviation  from  the  mean — content  analysis  not  required. 

Aircraft  Maintenance  Scheduling  and  Execution 
Filtering  Process:  Four  metrics  met  the  inclusion  criteria  for  the  Aircraft 
Maintenance  Scheduling  Process:  Aircraft  Availability,  Delayed  Discrepancy  Rate, 
Maintenance  Scheduling  Effectiveness,  and  Phase/Isochronal  Inspection  (Figure  23).  Six 
metrics  met  the  inclusion  criteria  for  the  Aircraft  Maintenance  Execution  Process:  Abort 
Rate,  Break  Rate,  Fix  Rate,  Maintenance  Scheduling  Effectiveness,  Mission  Capable 
Rate,  and  Repeat/Recur  Rate  (Figure  24).  Respondents  perceived  these  metrics  the  most 
relevant  to  the  Aircraft  Maintenance  Scheduling  and  Execution  Processes  and  should  be 
considered  as  part  of  the  Maintenance  Group’s  Balanced  Scorecard. 
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Figure  23:  Metric  Assignment  to  Aircraft  Maintenance  Scheduling  Process 
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Figure  24:  Metric  Assignment  to  Aircraft  Maintenance  Execution  Process 


Respondent  analysis  of  adequacy:  The  response  rate  met  both  the  50%  agreement 


rate,  and  exceeded  one  standard  deviation  from  the  mean  responses  and  indicated  79%  of 


respondents  agreed  there  were  adequate  metrics  for  Aircraft  Maintenance  and  Execution 


(Figure  25). 
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Figure  25:  Adequacy  of  Available  Metrics  for  Aircraft  Maintenance  Scheduling  Process 


Respondents  felt  measures  were  adequate,  and  at  least  one  measure  fell  above  one 
standard  deviation  from  the  mean — content  analysis  not  required. 

Equipment  Maintenance  Scheduling  and  Execution 
Filtering  Process:  Only  one  metric,  Maintenance  Scheduling  Effectiveness,  met 
the  inclusion  criteria  for  the  Equipment  Maintenance  Scheduling  Process  (Figure  26). 

The  filtering  process  for  the  Equipment  Maintenance  Scheduling  and  Execution  indicates 
that,  although  equipment  maintenance  may  be  important,  the  relevance  of  the  metrics  is 
insufficient  to  be  included  on  a  balanced  scorecard  (Figure  27). 
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Figure  27:  Metric  Assignment  to  Equipment  Maintenance  Execution  Process 


Respondent  analysis  of  adequacy:  The  response  rate  did  not  meet  the  50% 


agreement  rate  for  either  process  (Figure  28). 
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Figure  28:  Adequacy  of  Available  Metrics  for  Equipment  Maintenance  Scheduling 

Process 


Respondents  felt  measures  were  inadequate,  and  at  least  one  measure  fell  above 
one  standard  deviation  from  the  mean— content  analysis  was  required. 

Trend  issues  with  available  metrics:  Content  analysis  of  issues  respondents  had 
with  metrics  representing  this  process  indicated  that  metrics  available  for  measuring 
efficiency  and  effectiveness  of  equipment  maintenance  were  generally  inadequate,  or 
didn’t  exist  at  all,  with  many  suggestions  for  improvement  (Figure  29) 
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Figure  29:  Equipment  Maintenance  Scheduling  and  Execution  Process  Metrics  Issues 


Trend  suggested  metrics:  Upon  studying  open-ended  responses  that  further 
develop  respondents’  opinions  that  the  existing  metrics  for  Equipment  Maintenance 
Scheduling  and  Execution  are  inadequate,  several  metrics  suggestions  were  found  to  be  a 
trend.  Many  respondents  felt  equipment  maintenance  metrics  that  paralleled  many 
metrics  already  used  in  Aircraft  Maintenance  Scheduling  and  Execution  would  be  helpful 
in  studying  whether  the  processes  are  efficient  and  effective.  Equipment  Maintenance 
Scheduling  Effectiveness  Rate,  Break  Rates,  and  Fix  Rates  were  among  metrics 
suggested  by  Maintenance  Officers.  Additionally,  comments  suggested  that  it  may  be 
helpful  to  highlight  metrics  that  are  mandated  to  be  studied  at  the  flight  level  and  provide 
visibility  of  these  processes  at  the  Group  level.  Respondents  expressed  the  most  interest 
in  highlighting  Equipment  Mission  Capable  Rate  (or  In-Commission  Rate)  and 
Equipment  Availability  Rate  (Figure  30). 
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Equipment  Mx  Scheduling  and  Execution: 
Trended  Proposed  Metrics 


Repair  costs  I 


Figure  30:  Equipment  Maintenance  Scheduling  and  Execution  Process  Suggested  Metrics 

Readiness  Perspective 

Aircraft,  Maintainer  and  Equipment  Readiness 
Filtering  Process:  Five  metrics  met  the  inclusion  criteria  for  the  Aircraft 
Readiness  Category:  Abort  Rate,  Aircraft  Availability,  Break  Rate,  Delayed  Discrepancy 
Rate,  and  Mission  Capable  Rate  (Figure  3 1).  For  Maintainer  Readiness,  six  metrics  met 
the  inclusion  criteria:  CDC  Pass  Rate,  Manning  Rate,  QA  Pass  Rates,  QA 
TDV/DSV/UCR  Rates,  Training  Rate,  and  Upgrade  Training  Rate  (Figure  32).  For 
Equipment  Readiness,  one  metric  met  the  inclusion  criteria:  Spare  Engine  Status  (Figure 
33).  Respondents  perceived  these  metrics  the  most  relevant  to  Aircraft,  Maintainer,  and 
Equipment  Readiness  categories  and  should  be  considered  as  part  of  the  Maintenance 
Group’s  Balanced  Scorecard. 
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Figure  3 1 :  Metric  Assignment  to  Aircraft  Readiness 
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Figure  33:  Metric  Assignment  to  Equipment  Readiness 


Respondent  analysis  of  adequacy:  Enough  respondents  felt  selected  measures 
were  adequate  to  meet  the  50%  minimum  threshold  and  the  1  standard  deviation  from  the 
mean  threshold  for  Aircraft,  Maintainer  and  Equipment  Readiness  (Figure  34). 


Figure  34:  Adequacy  of  Available  Metrics  for  Equipment  Readiness 
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Respondents  felt  measures  were  adequate,  and  two  measures  fell  above  one 
standard  deviation  from  the  mean — content  analysis  was  not  required. 

Workforce  Quality  Perspective 
Maintenance  Quality  and  Safety 

Filtering  Process:  For  Maintenance  Quality,  seven  metrics  met  the  inclusion 
criteria  of  50%  selection  rate  and  were  above  one  standard  deviation  from  the  mean, 
including  Abort  Rate,  Break  Rate,  Delayed  Discrepancy  Rate,  Fix  Rate,  QA  Pass  Rates, 
QA  TDV/UCR/DSV  Rates,  and  Repeat/Recur  Rate  (Figure  35).  Maintenance  Safety  had 
two  metrics  meet  the  inclusion  criteria:  QA  Pass  Rate  and  QA  TDV/UCR/DSV  Rate 
(Figure  36).  Respondents  perceived  these  metrics  the  most  relevant  to  Maintenance 
Quality  and  Safety  and  should  be  considered  as  part  of  the  Maintenance  Group’s 
Balanced  Scorecard. 


Figure  35:  Metric  Assignment  to  Maintenance  Quality 
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Figure  36:  Metric  Assignment  to  Maintenance  Safety 


Respondent  analysis  of  adequacy:  Enough  respondents  felt  selected  measures 
were  adequate  to  meet  the  inclusion  criteria  (50%  minimum  threshold  and  the  1  standard 
deviation  from  the  mean  threshold)  (Figure  37). 
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Figure  37:  Adequacy  of  Available  Metrics  for  Maintenance  Quality  and  Safety 
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Respondents  felt  measures  were  inadequate,  and  no  measures  fell  above  one 
standard  deviation  from  the  mean — content  analysis  was  not  required. 

Maintainer  Skill  Level  and  Maintainer  Experience  Level 
Filtering  Process:  Four  metrics  met  the  inclusion  criteria  for  Maintainer  Skill 
Level:  CDC  Pass  Rates,  QA  Pass  Rates,  QA  TDV/UCR/DSV  Pass  Rates,  and  Upgrade 
Training  Rate  (Figure  38).  Maintainer  Experience  Level  had  four  metrics  meet  the 
inclusion  criteria:  Fix  Rate,  QA  Pass  Rate,  QA  TDV/UCR/DSV  Rates,  and  Repeat/Recur 
Rate  (Figure  39).  Respondents  indicated  these  metrics  are  the  most  relevant  to 
Maintainer  Skill  Level  and  Maintainer  Experience  Level  and  should  be  included  in  the 
Maintenance  Group’s  Balanced  Scorecard. 


Figure  38:  Metric  Assignment  to  Maintainer  Skill  Level 
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Figure  39:  Metric  Assignment  to  Maintainer  Experience  Level 


Respondent  analysis  of  adequacy:  Enough  respondents  felt  selected  measures 
were  adequate  to  meet  the  50%  minimum  threshold  and  the  1  standard  deviation  from  the 
mean  threshold  (Figure  40). 
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Figure  40:  Adequacy  of  Available  Metrics  for  Maintainer  Skill  and  Experience  Level 


Respondents  felt  measures  were  adequate,  and  four  measures  fell  above  one 
standard  deviation  from  the  mean — content  analysis  was  not  required. 
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Maintainer  Discipline  and  Maintainer  Retention 
Filtering  Process:  When  respondents  considered  metrics  for  Maintainer 
Discipline,  four  metrics  met  the  inclusion  criteria:  QA  Pass  Rates,  QA  TDV/UCR/DSV 
Rates,  Repeat/Recur  Rate  and  Training  Rate  (Figure  41).  Zero  metrics  met  the  50% 


selection  rate  for  Maintainer  Retention  (Figure  42). 
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Figure  4 1 :  Metric  Assignment  to  Maintainer  Discipline 


Figure  42:  Metric  Assignment  to  Maintainer  Retention 
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Respondent  analysis  of  adequacy:  When  considering  the  adequacy  of  available 
metrics,  responses  indicate  less  than  50%  felt  the  available  measures  were  adequate 
(Figure  43). 


Me,a"  28'4%V  47%  Maintainer  Disciplineand  Retention 

a=  18.6%  r 

100.0% 

90.0% 

80.0% 

70.0% 

60.0% 

50.0% 

40.0% 

30.0% 

20.0% 

10.0% 

0.0% 


Figure  43:  Adequacy  of  Available  Metrics  for  Maintainer  Discipline  and  Retention 


Respondents  felt  measures  were  inadequate  when  asked  generally  about  both 
categories;  however,  Maintainer  Retention  Category  had  zero  metrics  that  met  the  criteria 
for  inclusion.  Therefore,  content  analysis  was  required,  and  focused  on  the  Maintainer 
Retention  Category. 

Trend  issues  with  available  metrics:  When  the  researcher  reviewed  the 
respondents’  comments  there  was  a  general  feel  that  retention  is  a  concern  for  the 
Maintenance  Community,  but  had  a  wide  spread  of  views  on  how  to  measure  it. 
Additionally,  factors  such  as  sequestration,  force  shaping,  voluntary  separation  programs, 
and  involuntary  separations  make  maintainer  retention  a  fluid  situation  that  would  be 
difficult  to  meaningfully  measure  at  the  Group  level  (Figure  44). 
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Figure  44:  Maintainer  Retention  Metrics  Issues 


Trend  suggested  metrics:  When  asked  to  suggest  metrics  that  could  better 
articulate  Maintainer  Retention,  most  respondents  believed  a  re-enlistment  rate  would  be 
beneficial  to  the  Maintenance  Group  as  a  lagging  indicator.  The  other  trend  amongst 
comments  were  to  track  items  that  that  they  believed  contributed  to  job  dissatisfaction, 
which  could  be  correlated  to  retention  rate— Weekend  Duty  Rate,  Overtime  Rate  were  the 
most  commonly  suggested  metrics  as  more  “leading  indicators”  to  the  re-enlistment  rate 
“lagging  indicator”  (Figure  45). 
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Maintainer  Discipline  and  Retention: 
Trended  Proposed  Metrics 
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Figure  45:  Maintainer  Retention  Suggested  Metrics 


Balanced  Scorecard  utility  and  frequency 

The  purpose  of  the  next  section  of  the  survey  was  to  determine  the  utility  of  the 
balanced  scorecard  framework  in  Maintenance  Group.  If  deemed  beneficial  by 
Maintenance  experts  surveyed,  the  next  step  would  be  to  detennine  the  optimal  frequency 
of  analysis  of  the  metrics  in  each  perspective.  The  survey  asked  respondents  to  assess  the 
benefit  of  examining  metrics  representing  each  perspective  in  the  same  meeting,  in  order 
for  the  researcher  to  detennine  field  support  or  opposition  of  the  concept.  The  responses 
indicated  that  60%  of  the  surveyed  population  Agreed  or  Strongly  Agreed  that  the 
balanced  scorecard  concept  would  be  beneficial  to  study  metrics  from  each  perspective  in 
the  same  meeting  (Figure  46). 
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Figure  46:  Balanced  Scorecard  Utility  Perception 


The  next  question  asked  respondents  to  indicate  at  what  management  level  and  at 
what  frequency  the  balanced  scorecard  framework  should  be  studied  to  gain  optimal 
utility  and  understanding  from  the  metrics. 

Filtering  Process:  Respondents  felt  that  the  balanced  scorecard  framework  would 
have  most  utility  when  studied  at  the  Wing  Level  quarterly,  and  at  the  Group  Level 
monthly.  Responses  also  indicated  preferred  frequencies  for  Squadron  level  and  flight 
level  review;  however,  these  responses  did  not  meet  the  50%  minimum  threshold  of 
responses  (Figure  47). 
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Figure  47:  Balanced  Scorecard  Frequency  Recommendations 


Efficacy  of  Individual  Metrics 

Filtering  Process:  The  last  section  of  the  survey  asked  respondents  to  evaluate  28 
individual  maintenance  metrics  based  on  their  experience  utilizing  and  analyzing  metrics. 
The  survey  asked  respondents  to  select  every  metric  they  believe  is  not  "good"  for  any 
reason,  and  then  asked  them  to  explain  their  reason  for  each  metric  selected.  Aircraft 
inventory,  Fix  Rate,  and  Workcenter  Utilization  rate  fell  above  one  standard  deviation 
from  the  mean  number  of  responses,  and  therefore  were  flagged  as  suspect  metrics 
(Figure  48). 
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Figure  48:  Individual  Metrics  Efficacy  Response 


Additionally,  the  researcher  found  that  for  the  19  perspective  categories  outlined 
in  this  study,  five  metrics  were  not  associated  with  any  perspective  category  in  the 
Appendix  H  cross  tabulation  because  they  did  not  meet  the  50%  minimum  response 
threshold  in  addition  to  the  one  standard  deviation  standard.  Between  the  responses  to 
survey  question  evaluating  efficacy  of  individual  metrics,  and  the  actual  assignment  of 
metrics  to  perspective  categories,  the  seven  metrics  identified  for  Content  Analysis  were: 

Aircraft  Inventory 

The  responses  involving  Aircraft  Inventory  (Figure  49)  were  that  this  metric  did 
not  drive  action,  as  the  number  of  aircraft  assigned  to  wings  was  beyond  the  Maintenance 
Group’s  control. 
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Aircraft  Inventory  (Backup  (BAI)  and 
Primary(PAI)) 


Figure  49:  Aircraft  Inventory  Efficacy  Evaluation 


Fix  Rate 

The  Fix  Rate  (Figure  50),  of  all  the  metrics  that  directly  measure  aircraft 
maintenance  actions,  was  viewed  by  respondents  to  be  of  the  least  utility  for  multiple 
reasons.  Comments  regarding  the  Fix  Rate  largely  addressed  the  rate’s  lack  of  behavioral 
soundness  because  of  its  propensity  to  incite  behavior  counterproductive  to  sound  aircraft 
maintenance  practices  and  quality  aircraft  repairs.  Respondents’  comments  indicate  that 
emphasis  on  the  Fix  Rate  causes  rushed  maintenance  action  and  a  subsequent  rise  in  the 
Repeat/Recur  Rate.  One  officer  stated  that  the  Fix  Rate  is  “counter  to  every  other  tenet 
of  aircraft  maintenance  (safe,  reliable,  and  by  the  book).”  Additionally,  many 
respondents  felt  the  4-,  8-  or  12-hour  standards  for  this  rate  were  arbitrary,  antiquated  and 
relatively  useless  in  modem  day  aircraft  maintenance  where  aging  aircraft  require  more 
time  to  fix  and  the  newest  5th  generation  aircraft  can’t  possibly  achieve  the  standard  when 
perfonning  low  observable  (LO)  intrusive  repairs.  Comments  indicate  that  a  more 
relevant  measure  of  repair  efficiency  would  be  a  metric  that  measures  completion  times 
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as  compared  to  a  job  standard  repair  time  and  that  encompassed  all  maintenance  actions 
rather  than  just  pilot  reported  discrepancies.  Without  established  job  standard  repair 
times,  many  felt  the  Fix  Rate  lacked  sufficient  level  of  detail  and  was  considered  benign. 


Fix  Rate  (4/8/12-hour) 

Behavorial  Soundess 
Level  of  Detail 
Compatability 
Economy 
Integration 
Usefullness 
Robustness 
Validity 

0.0%  20.0%  40.0%  60.0%  80.0%  100.0% 


Figure  50:  Fix  Rate  Efficacy  Evaluation 


Workcenter  Utilization  Rate 

The  researcher  noticed  that  responses  trending  around  Workcenter  Utilization 
Rate  (Figure  5  ljwere  that  the  data  input  into  the  information  system,  IMDS,  was  not 
reliable  and  therefore  this  metric  was  not  of  use  to  them. 


Figure  5 1 :  Workcenter  Utilization  Rate  Efficacy  Evaluation 
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Aircraft  Possessed 


The  responses  involving  Aircraft  Possessed  (Figure  52)  were  very  similar  to 
comments  regarding  Aircraft  Inventory;  this  metric  did  not  drive  action,  as  the  number  of 
aircraft  possessed  to  wings  was  largely  beyond  the  Maintenance  Group’s  control  and  was 
generally  used  as  part  of  calculations  in  other,  controllable  metrics. 


Aircraft  Possessed  (Average,  Primary 
Possessed  Rate) 


Figure  52:  Aircraft  Possessed  Efficacy  Evaluation 


Cannibalization  Rate 

The  number  of  responses  assigning  this  metric  to  the  19  perspective  categories  did 
not  meet  the  filtering  criteria  for  representing  any.  Respondents  who  commented  on  the 
Cannibalization  Rate  (Figure  53)  felt  it  that  it  provided  a  guide  for  action  not  for  the 
Maintenance  Group,  but  for  Supply  and  the  Mission  Support  Group.  One  respondent 
commented,  “MICAP/CANN  or  supply  effectiveness  rates  are  only  good  if  they  generate 
an  improvement  in  the  overall  contract/supply  process...  the  end  user  (Aircraft 
Maintenance  Unit  OIC)  is  usually  the  one  briefing  the  MICAP  or  CANN  rate,  instead  of 
an  LRS  officer  being  required  to  brief  the  MICAP/CANN/Supply  Effectiveness  rates  to 
the  AMU  OIC.” 
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Cannibalization  Rate 
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Figure  53:  Cannibalization  Rate  Efficacy  Evaluation 


Mission  Impaired  Capability  (MICAP) 

Similar  to  the  Cannibalization  rate,  the  MICAP  Rate  (Figure  54)  did  not  meet  the 
filtering  criteria  to  be  considered  in  any  of  the  19  perspective  categories.  All  respondent 
comments  indicated  the  MICAP  rate  was  a  supply  metric  and  maintenance  had  no 
influence  over  the  supply  chain  and  subsequently  could  not  influence  the  MICAP  Rate. 
When  considering  metrics  regarding  supply  issues,  maintainers  find  much  more  utility  in 
the  Non-Mission  Capable  for  Supply  (NMCS)  Rate  because  it  shows  most  clearly  the 
effect  of  supply  on  combat  capability.  As  for  the  MICAP  rate,  respondents  rated  its 
usefulness  minimal  at  best. 
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Mission-Impaired  Capability  Awaiting  Parts 
(MICAP) 


Figure  54:  Mission-Impaired  Capability  Rate  Efficacy  Evaluation 

Supply  Effectiveness  Rate 

As  with  other  supply  chain  oriented  metrics,  respondents  felt  the  Supply 
Effectiveness  Rate  (Figure  55)  was  not  worthy  of  consideration  to  be  represented  in  any 
of  the  19  perspective  categories.  Citing  an  inability  to  influence  the  supply  chain  or  the 
metric,  the  Supply  Effectiveness  rate  may  be  of  use  to  supply  analysts  in  the  Support 
Group,  but  it  has  nearly  no  utility  being  tracked  as  a  metric  for  the  Maintenance  Group. 
Once  again,  the  NMCS  Rates  is  far  more  useful  to  maintainers  because  it  clearly  shows 
the  impact  to  combat  capability. 


too 


Supply  Effectiveness  Rate 


Figure  55:  Supply  Effectiveness  Rate  Efficacy  Evaluation 

Maintenance  Group  Balanced  Scorecard  Initial  Rollup 

Using  the  list  of  metrics  that  met  the  criteria  for  inclusion  under  each  category, 
combined  with  the  evaluation  of  individual  metrics  to  ensure  metrics  included  are  sound, 
the  researcher  developed  the  initial  Maintenance  Group  Balanced  Scorecard  shown  in 
Figure  56.  The  researcher  will  use  contents  of  this  initial  version  to  perform  a 
Correlation  Analysis  to  further  adjust,  refine,  and  produce  the  final  proposal  for  the 
Maintenance  Group  Balanced  Scorecard. 
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Figure  56:  Initial  Maintenance  Group  Balanced  Scorecard 


Correlation  Analysis 

Perspective  Category  Correlation 

The  perspective  category  combinations  and  their  correlation  coefficients  are 
outlined  in  Appendix  I.  The  researcher  found  that  many  of  the  high  correlations  occurred 
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were  due  to  corresponding  low  selection  rates  for  metrics  assignments  to  perspective 
categories.  Even  though  high  correlation  coefficients  were  observed  in  many  cases,  the 
metrics  that  were  assigned  to  process  categories  weren’t  always  the  metrics  that  caused 
high  correlations.  However,  in  several  cases,  metrics  that  were  in  the  top  three  or  four 
selections  of  perspective  categories  were  highly  correlated  between  perspective 
categories  and  worthy  of  analysis  and  discussion.  During  correlation  analysis,  the 
researcher  identified  several  areas  of  concern,  where  metrics  assigned  to  different 
perspective  categories  overlapped  excessively. 

The  first  area  of  concern  highlighted  by  high  correlations  was  between  Maintainer 
Training  Scheduling  and  Execution,  Maintainer  Readiness,  and  all  Workforce  Quality 
Perspective  categories.  QA  Pass  rates,  QA  STV/TDV/UCR  rates,  Training  rates, 

Upgrade  Training  Rates,  Repeat/Recur  Rates  and  CDC  Pass  Rates  were  highly  correlated 
between  all  of  these  categories.  Since  Maintenance  Quality,  Maintenance  Safety,  and 
Maintenance  Discipline  are  all  measured  during  QA  inspections  and  represented  in 
specific  QA  rates,  the  researcher  elected  to  combine  these  three  Workforce  Quality 
Perspective  categories  into  one  category  called  Maintenance  Quality,  Safety  and 
Discipline,  and  eliminate  those  metrics  from  being  represented  among  the  other 
categories. 

Maintainer  Experience 

Metrics  selected  for  Skill  Level  and  Experience  Level  under  the  Workforce 
Quality  Perspective  were  very  closely  correlated,  concerning  the  researcher  and  forcing 
closer  examination  through  Content  Analysis.  As  described  in  Chapter  II,  skill  level  is 
awarded  after  completion  of  a  set  of  tasks  in  the  CFETP;  however,  a  technician’s  skill 
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level  does  not  necessarily  equate  to  experience  level  or  expertise.  The  problem  with  skill 
level  is,  once  achieved,  it  remains  with  an  Airman  regardless  of  assignment  or  type  of 
airframe.  For  example,  a  Staff  Sergeant  that  earns  his  7-level  while  working  and  training 
on  the  F-15  may  be  considered  a  craftsman  of  substantial  skill  because  of  his  experience, 
until  he  is  assigned  to  an  A- 10  unit  in  Korea  and  finds  himself  at  a  disadvantage  because 
of  his  lack  of  experience  on  the  dissimilar  airframe.  The  simple  fact  of  the  matter  is, 
experience  level  on  an  airframe  matters  and  it  cannot  be  equated  with  skill  level.  An 
Ainnan  with  4  years  experience  on  an  airframe,  although  only  a  5 -level,  is  most  often 
more  capable  than  a  7-level  with  only  3  months  on  that  same  airframe.  MDS  experience 
level  has  become  increasingly  relevant  in  recent  years  as  corporate  knowledge  on  the 
flightline  dwindles  with  the  manning  cuts.  The  researcher  elected  to  perform  additional 
content  analysis  on  comments  between  these  two  perspectives  to  investigate  whether  the 
high  correlation  was  due  to  redundant  categories,  or  if  metrics  were  lacking. 

Trend  issues  with  available  metrics:  During  content  analysis,  the  evidence  was 
overwhelming  that  a  lack  of  metrics  was  the  culprit  for  the  high  correlation.  Over  60%  of 
the  respondents  assessed  that  there  were  adequate  metrics  to  measure  skill  level,  but 
inadequate  metrics  to  measure  experience  level  of  the  workforce  (Figure  57). 
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Maintainer  Experience: 
Issues  with  Available  Metrics 


should  be  looked  at  MXG  external  internal  are  inadequate  are  adequate 

level  but  not  sources  but  sources  but 
listed  as  not  visible  to  not  looked  at 
choices  MXG  MXG  level 


Figure  57:  Experience  Level  Metrics  Issues 


Trend  suggested  metrics:  The  majority  of  respondents  believed  that  a  metric 
measuring  Experience  Level,  especially  with  looming  manning  cuts,  is  required  for  units 
to  truly  understand  their  capabilities.  Many  suggested  that  measuring  time  on  an  MDS  is 
a  good  place  to  start  measuring  experience  level.  Skill  Experience  Identifiers  are  meant 
to  measure  this,  but  many  respondents  indicated  this  process  was  “broken”  and  not 
meaningful.  Overall,  respondents  felt  it  was  critical  to  separate  skill  level  from 
experience  level  and  should  be  given  as  much,  if  not  more,  consideration  as  skill  levels 
are  given  when  assessing  manning  situations.  For  this  reason,  other  metrics  were 
eliminated  from  this  category  and  replaced  with  “experience  level”  as  a  more  direct 
measure  (Figure  58). 
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Maintainer  Experience: 
Trended  Proposed  Metrics 


Figure  58:  Experience  Level  Suggested  Metrics 


Maintainer  Readiness 

The  researcher  explored  the  issue  of  correlation  in  the  Maintainer  Readiness 
category  next.  The  intent  of  readiness  categories  is  to  give  leaders  and  commanders  a 
bottom-line  view  of  the  readiness  of  aircraft,  equipment,  and  personnel;  however, 
Maintainer  Readiness  is  currently  being  measured  with  six  different  metrics,  all  which 
could  be  considered  “leading  indicators”  versus  the  “lagging  indicator  that  is  meant  to  be 
shown  in  this  perspective.  Additionally,  the  assigned  metrics  were  used  to  measure  other 
categories,  specifically,  the  Maintenance  Training  and  Execution  and  Workforce  Quality 
categories.  The  researcher  opted  to  perfonn  additional  content  analysis  on  Readiness 
Rates  to  understand  the  context  of  selections. 

Trend  issues  with  available  metrics  for  Maintainer  Readiness:  As  depicted  in 
Figure  59,  respondents  felt  the  main  issues  with  metrics  measuring  Maintainer  Readiness 
were  that  they  weren’t  available,  or  metrics  that  were  available  were  inadequate. 
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Generally,  the  responses  indicated  the  inadequacy  meant  the  available  metrics  only 
indirectly  painted  the  picture  of  Maintainer  Readiness— Maintainer  Readiness  had  to  be 
inferred  based  on  maintenance  performance  metrics  or  training  metrics. 


Maintainer  Readiness: 
Trended  Issues  with  Available  Metrics 


100.0% 

90.0% 

80.0% 

70.0% 

60.0% 

50.0% 

40.0% 

30.0% 

20.0% 

10.0% 

0.0% 


Not  Don't  need  to  No  metrics  Metrics  Metrics  Metrics  Metrics  Metrics 
categorizable  track  available  but  available  and  Available  available  available  but  available  and 
should  be  lookedat  from  external  from  internal  are  areadequate 
MXG  level  but  sources  but  sources  but  inadequate 
not  listed  as  not  visible  to  not  looked  at 
choices  MXG  MXG  level 


Figure  59:  Maintainer  Readiness  Metrics  Issues 


Trend  suggested  Maintainer  Readiness  metrics:  Suggestions  for  better  metrics 
were  numerous  and  diverse,  but  similar  to  metrics  assigned  to  Aircraft  Readiness 
category,  Capability  and  Availability  were  among  the  suggestions  (Figure  60).  Some 
respondents  suggested  that  a  Capability  Metric  be  rolled  up  from  Skill  levels  and 
Experience  levels,  and  availability  be  represented  by  the  number  of  Airmen,  who  fill  the 
proper  manning  slot  of  unit  manning  documents,  and  are  available  for  duty.  For  this 
reason,  the  researcher  suggested  Maintainer  Capability  and  Maintainer  Availability  for 
the  Maintainer  Readiness  category,  and  moved  other  “leading  indicators”  to  either 
Training  Scheduling  or  Execution  Category,  or  to  the  Maintenance  Quality,  Safety  and 
Discipline  Category.  Additionally,  several  respondents  mentioned  duty  hours  as  factors 
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as  well  as  resiliency  indicators  were  suggested  as  well;  as  these  types  of  metrics  were 
already  largely  categorized  under  Maintainer  Retention,  the  researcher  elected  to  include 
those  responses  under  that  category 


Figure  60:  Maintainer  Readiness  Proposed  Metrics 


Maintenance  Training  and  Execution  Process 

This  left  Maintenance  Training  and  Execution  as  the  final  area  to  be  addressed 
under  the  first  set  of  correlation  concerns.  As  Maintenance  Training  and  Execution 
processes  seem  to  be  well  represented  with  the  leading  indicators  of  Career  Development 
Course  Success  Rate,  Training  Rate,  and  Upgrade  Training  Rate,  the  researcher  elected  to 
remove  those  metrics  from  all  other  categories  and  solely  represent  them  in  the 
Maintenance  Training  and  Execution  Process  category. 
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Aircrew  Mission/Training  Scheduling  and  Execution  Process  and 
Flying  Scheduling  and  Execution 

The  next  area  of  concern  fell  between  the  Aircrew  Mission/Training  Scheduling 
and  Execution  Processes  and  the  Flying  Scheduling  and  Execution  Processes.  The 
researcher  detennined  that  the  high  correlation  between  those  two  categories  was  a  result 
of  lack  of  better  metrics  to  measure  the  former.  Aircrew  Mission/Training  Scheduling 
and  Execution  Processes  refer  specifically  to  the  efficiency  and  effectiveness  of  Aircrew 
Training  requirements.  Flying  Scheduling  Effectiveness  refers  to  take  off  and  land  times, 
and  deviations  from  the  flying  schedule,  with  little  consideration  for  deviations  from 
planned,  briefed  and  flown  Missions  that  Aircrew  are  required  to  accomplish.  Abort  rate 
would  logically,  albeit  loosely,  tie  these  processes  together,  but  an  abort  doesn’t 
necessarily  mean  a  spare  wasn’t  provided  and  the  mission  was  accomplished  as  planned. 
As  detennined  in  the  Content  Analysis  of  Aircrew  Mission/Training  Scheduling  and 
Execution,  this  conelation  corroborated  the  need  for  different  and  unique  metrics  to 
properly  measure  efficiency  and  effectiveness  of  this  process,  which  are  the  RAP/Mission 
Effectiveness  Rate  and  Mission  Planning  Deviation  rates. 

Equipment  Maintenance  Scheduling  and  Execution  and  Equipment 
Readiness 

Another  area  of  concern  was  the  high  correlation  between  Equipment 
Maintenance  Scheduling  and  Execution,  and  Equipment  Readiness.  As  pointed  out  in  the 
Content  Analysis  of  Maintainer  Readiness,  the  intent  of  readiness  categories  is  to  give 
leaders  and  commanders  a  bottom-line  view  of  the  readiness  of  aircraft,  equipment,  and 
personnel;  however,  two  of  the  metrics  in  the  Equipment  Maintenance  Scheduling  and 
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Execution  category  are  considered  “lagging  indicators”,  which  are  more  appropriate  for 
the  Equipment  Readiness  category.  The  researcher  opted  to  perfonn  additional  content 
analysis  on  Readiness  Rates  to  understand  the  context  of  selections. 

Trend  issues  with  available  metrics  for  Equipment  Readiness:  As  depicted  in 
Figure  61,  respondents  felt  the  main  issues  with  metrics  measuring  Equipment  Readiness 
were  that  they  weren’t  available,  or  metrics  that  were  available  were  inadequate. 
Generally,  the  responses  indicated  the  inadequacy  meant  the  available  metrics  only 
indirectly  painted  the  picture  of  Equipment  Readiness— Equipment  Readiness  only  had 
Spare  Engine  status  to  represent  it,  currently,  while  there  are  many  more  pieces  of 


equipment  in  which  Readiness  is  also  critical. 
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Figure  61 :  Equipment  Readiness  Issues  with  Available  Metrics 


Trend  suggested  Equipment  Readiness  metrics:  Suggestions  for  better  metrics 


were  straightforward.  Similar  to  metrics  assigned  to  Aircraft  Readiness  category,  and 
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proposed  for  Maintainer  Readiness  category,  Mission  Capability,  In-commission  Rates 
and  Equipment  Availability  were  the  top  suggestions  (Figure  62).  The  researcher  moved 
the  only  “leading”  metric  currently  listed  in  the  category,  repair  rates,  to  the  Equipment 
Maintenance  Scheduling  and  Category  to  maintain  the  integrity  of  the  “bottom-line 
indicators”  of  the  Equipment  Readiness  category. 


Figure  62:  Equipment  Readiness  Proposed  Metrics 


Aircraft  Scheduling  and  Execution  and  Equipment  Maintenance 
Scheduling  and  Execution 

Another  area  of  concern  during  this  correlation  analysis  was  between  the  Aircraft 
Scheduling  and  Execution  Processes,  and  the  Equipment  Maintenance  Scheduling  and 
Execution  Processes.  Considering  that  no  Equipment-specific  measures  were  listed,  there 
should  have  been  very  low  correlation  between  these  four  process  categories.  The 
researcher  felt  it  was  safe  to  assume  that  the  respondents'  intent  when  assigning  these 
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aircraft-specific  metrics  such  as  MSE,  Fix  Rate,  Break  Rate,  and  Mission  Capable  Rate 
were  meant  to  be  equipment-related  metrics.  Equipment  MSE,  Equipment  Break  Rate, 
and  Equipment  Repeat/Recur  Rate  should  be  considered  different  metrics  from  their 
aircraft  counterparts  and  included  in  the  Equipment  Maintenance  Execution  category. 

Scheduling  and  Execution 

The  last  area  of  concern  was  between  Scheduling  and  Execution  Processes  in 
general.  While  efficiency  and  effectiveness  in  a  Scheduling  process  could  theoretically 
be  measured  differently  than  efficiency  and  effectiveness  in  an  Execution  Process,  the 
reality  of  measuring  them  in  a  unique  manner  from  each  other  makes  a  case  for 
combining  those  measures  together.  How  else  can  one  measure  efficiency  in  scheduling, 
except  to  see  the  fruits  of  the  execution  of  the  schedule?  An  effective  schedule  means 
fewer  changes  to  the  schedule  during  execution.  An  efficient  scheduling  process  could 
measure  the  man-hours  it  takes  to  route  a  schedule  through  the  chain  of  command  for 
approval,  but  economy  of  effort  to  capture  that  data  could  be  focused  elsewhere.  For  this 
reason,  the  researcher  elected  to  combine  scheduling  and  execution  for  all  of  the  process 
perspective  categories. 

After  combining  the  Scheduling  and  Execution  Process  categories,  the  researcher 
noticed  the  top  metrics  for  Aircraft  Maintenance  Scheduling  and  Execution  reflected  two 
different  actions— scheduled  and  unscheduled  maintenance.  As  such,  the  efficiency  and 
effectiveness  of  execution  of  scheduled  and  unscheduled  maintenance  actions  should  be 
measured  with  different  yardsticks.  Unscheduled  maintenance  actions  are  just  that— 
unscheduled,  so  the  MSE  rate,  while  highly  telling  of  the  efficiency  of  scheduled 


112 


maintenance,  is  not  applicable.  The  researcher  elected  to  break  this  perspective  category 
into  Unscheduled  and  Scheduled  Maintenance  activities. 

Correlation  analysis  proved  to  be  extremely  useful,  even  after  rigorous  filtering 
and  content  analysis  performed  earlier  in  the  methodology.  First,  the  researcher  was  able 
to  identify  redundancies  among  category  intent.  Additionally,  the  researcher  was  able  to 
identify  where  respondents  assigned  metrics  to  categories  not  because  they  were  ideal, 
but  because  they  were  the  merely  the  best  they  had  to  choose  from.  Following  additional 
content  analysis  from  these  revelations,  the  researcher  felt  confident  that  the  revised 
Balanced  Scorecard  could  meet  the  needs  of  maintenance  leaders  and  decision  makers  in 
understanding  the  performance  in  processes  towards  meeting  customer  requirements, 
readiness,  and  workforce  quality  issues. 
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V.  Discussion,  Recommendations  and  Conclusions 


Discussion 

This  chapter  first  addresses  the  assumptions  and  limitations  identified  throughout 
the  research  process.  After  these  assumptions  are  addressed,  the  researcher  will  answer 
the  three  Research  Questions  outlined  in  Chapter  I  of  this  paper.  Implications  of  this 
research  will  then  be  covered,  followed  by  several  recommendations  for  future  research. 
Finally,  other  recommendations  will  be  presented 

Assumptions  and  Limitations 

To  support  this  study  of  the  utility  of  the  balanced  scorecard  framework  and 
utility  of  metrics  already  in  use  in  aircraft  maintenance,  the  researcher  conducted  a  survey 
to  capture  the  opinions  of  maintenance  officers  serving  in  the  CAF.  The  survey 
canvassed  as  many  officers  as  possible  to  ensure  a  well-rounded,  unbiased  perspective 
and  for  a  credible  reference  throughout  the  research  process.  The  researcher  must  assume 
that  the  responses  and  opinions  of  the  survey  respondents  are  an  accurate  representation 
of  the  opinions  of  the  entire  Aircraft  Maintenance  community. 

The  researcher  used  personal  judgment  to  interpret  the  meaning  of  the 
respondents’  comments  during  the  coding  process  of  Content  Analysis.  Although  an 
external  auditor  was  used  to  validate  the  results  of  the  coding  process,  it  must  be 
acknowledged  that  interpretation  of  comments  is  subject  to  error  when  clarification  or 
amplification  of  responses  is  not  possible.  The  researcher  assumed,  for  the  purpose  of  the 
research,  that  the  respondents’  comments  were  interpreted  correctly. 

The  researcher  investigated  multiple  criteria  to  assess  the  current  aircraft 
maintenance  metrics  set,  but  limited  the  discussion  to  the  criteria  presented  by  Caplice 
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and  Sheffi  based  on  her  personal  assessment  of  the  criteria  as  compared  to  others.  This 
study  assumes  the  author  was  objective  in  her  choice  of  criteria  and  did  indeed  choose  the 
most  comprehensive  and  complete  criteria  to  develop  the  survey. 

The  researcher  did  not  include  formulas  for  the  metrics  evaluated  in  the  survey, 
and  assumed  that  the  maintenance  officers  who  participated  in  the  survey  had  knowledge 
of  these  formulas  or  had  access  to  the  metrics  formulas  if  they  required  them. 

Implementation  of  the  Maintenance  Group  Balanced  Scorecard  framework  could 
be  limited  by  Department  of  Defense  classification  standards  for  readiness-related 
infonnation.  In  addition,  current  information  systems  may  not  be  equipped  to  capture 
required  data  to  implement  the  proposed  metrics  outlined  in  this  study. 

Recommendations 

The  researcher’s  investigation  revealed  infonnation  gaps  provided  by  the  current 
aircraft  maintenance  metrics  set  and  the  method  and  frequency  of  their  study,  and 
recommends  several  metrics  be  refined,  created,  or  eliminated  to  populate  the 
Maintenance  Group  Balanced  Scorecard,  as  indicated  by  respondents. 

Metrics  to  Create 

First,  the  researcher  recommends  the  current  Fix  Rate  be  refined  to  eliminate  the 
negative  behavior  described  by  respondents  before  incorporating  into  the  balanced 
scorecard.  The  researcher  recommends  the  length  of  time  it  takes  to  fix  a  particular 
discrepancy  be  measured  against  an  established  job  standard  time  versus  against  an 
arbitrary  4-,  8-,  or  12-hour  standard.  The  proposed  “Job  Standard  Fix  Rate”  would 
provide  maintainers  clear  expectations  of  performance  for  individual  task 
accomplishment  and  enable  more  accurate  measurement  of  efficiency  of  repairs  than  an 
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arbitrary  standard.  The  clarity  and  objectivity  provided  by  the  Job  Standard  Fix  Rate 
would  reduce  safety  and  quality  concerns  as  maintainers  would  be  less  likely  to  be 
pressured  by  an  unrealistic  expectation  of  their  work  tempo.  An  additional  benefit  of  the 
Job  Standard  Fix  Rate  would  be  a  clear  depiction  of  training  deficiencies  or  process 
inefficiencies  on  specific  tasks.  For  example,  if  engine  changes  consistently  exceeded  an 
established  job  standard  time,  managers  would  have  statistical  evidence  that  increased 
training  is  required  or  be  able  to  examine  the  process  to  eliminate  non-value  added 
elements  hindering  optimization.  The  Job  Standard  Fix  rate  would  be  useful  in  both  the 
internal  process  perspective  and  the  learning  and  growth  perspective  of  the  Maintenance 
Group’s  Balanced  Scorecard. 

Examination  of  survey  responses  regarding  Equipment  Maintenance  processes 
exposed  infonnation  gaps  and  opportunities  for  improvement  as  well.  The  researcher 
recommends  the  creation  of  metrics  to  measure  efficiency  and  effectiveness  of 
Equipment  Maintenance  Scheduling  and  Execution,  including  Equipment  In-commission 
Rate,  Equipment  Availability  Rate,  an  Equipment  Maintenance  Scheduling  Effectiveness 
Rate,  and  an  Equipment  Break  Rate. 

Another  valuable  discovery  of  this  research  was  the  need  for  increased  visibility 
of  Aircrew  Mission/Training  Scheduling  and  Execution  processes.  Although  Operations’ 
processes  do  not  fall  into  the  Maintenance  Group’s  sphere  of  control,  the  Maintenance 
Group  has  significant  impact  on  the  efficiency  and  effectiveness  of  those  processes. 
Survey  responses  indicate  that  increased  transparency  of  metrics  articulating  customer 
perspective,  and  specifically  the  Maintenance  Group’s  impact  on  them,  could  lead  to 
improved  efficiency  in  Maintenance  Group  processes.  Respondents  indicated  a  need  to 
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know  exactly  how  maintenance  operations  affect  Aircrew,  and  to  know  the  results  of 
what  they  were  providing  for  from  a  RAP  perspective.  These  metrics  include  Operations 
Scheduling  Effectiveness  Rate  (Missions  briefed  vs.  Missions  planned),  a  Mission 
Effectiveness  Rate  (Missions  flown  vs.  Mission  Planned),  and  a  RAP  Rate  (RAP  plan 
met  for  month/quarter).  The  researcher  received  several  personal  emails  from 
Maintenance  Officers  whose  organizations  were  already  pursuing  these  types  of  metrics 
and  have  established  programs  that  could  lay  the  groundwork  for  implementation  of  these 
metrics  CAF-wide. 

Furthennore,  the  researcher  recommends  metrics  to  close  gaps  in  the  bottom-line 
measures  of  maintenance  personnel  and  equipment  readiness.  Respondents  indicated  that 
measures  similar  to  those  studied  for  Aircraft,  such  as  Capability  and  Availability,  would 
be  helpful  in  decision  making  and  optimizing  the  impact  of  these  important  enablers  of 
airpower.  Respondents  suggested  that  the  Maintainer  Capability  measure  would  best 
serve  the  needs  of  managers  if  it  were  a  roll-up  of  skill  level  and  experience  level. 

The  development/study  of  Workforce  Quality  measures  is  imperative  to 
maximizing  the  contribution  and  effectiveness  of  the  increasingly  limited  workforce.  The 
researcher  recommends  Workforce  Quality  indicators  be  created  to  understand  the 
condition  of  the  enablers  of  the  mission-execution— the  maintainers.  Recommended 
measures  under  this  perspective  include  measures  that  could  affect  retention  such  as 
Overtime  Rate,  and  Weekend  Duty  Rate.  Also,  a  separate  evaluation  of  skill  level  and 
experience  level  is  recommended  under  this  perspective  as  one  of  the  unintended 
consequences  of  force  reduction  and  restructuring  is  that  skill  level  and  airframe 
experience  level  have  become  disparate. 
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Metrics  to  Reassess 


The  researcher  recommends  a  reassessment  of  Supply-related  metrics. 
Respondents  largely  felt  that  rates  such  as  MICAP  rate,  Cannibalization  Rate,  and  Supply 
Effectiveness  Rate  could  not  be  controlled  at  the  Maintenance  Group  level,  and  these 
metrics  would  be  more  useful  in  the  hands  of  members  of  the  Logistics  Squadrons.  As 
the  Proposed  Maintenance  Group  Balanced  Scorecard  did  not  include  this,  an  area  to  be 
explored  for  incorporation  into  the  balanced  scorecard  is  a  Supplier  Performance  Section, 
to  include  external  agencies  perfonnance  in  providing  to  the  Maintenance  Group. 

Scrutiny  of  Supplier  Performance  is  necessary  to  ensure  supply  enablers  remain  vested  in 
the  success  of  the  Maintenance  Group. 

The  researcher  recommends  reassessment  of  Workcenter  Utilization  rate,  how 
data  is  gathered  to  feed  this  metric,  and  what  this  metric  is  used  for.  In  an  environment  as 
dynamic  as  maintenance,  many  expressed  the  difficulty  in  obtaining  accurate  time  that 
maintainers  spent  on  a  job,  and  expressed  concern  that  inaccurate  data  is  being  used  to 
make  decisions  about  manning  and  force  structure.  It  is  important  to  note,  that  a 
reevaluated  and  improved  Workcenter  Utilization  Rate  considered  with  the  Job  Standard 
Fix  Rate  could  reveal  useful  information  regarding  Workforce  Quality  and  how 
efficiently  the  workforce  is  employed. 

Recommendation  Summary/Research  Question 
The  summary  of  the  preceding  recommendations  for  action  for  each  of  the  28 
metrics  evaluated,  as  well  as  recommendations  for  metrics  proposed  by  respondents,  are 
detailed  in  Figure  63,  and  illustrated  in  the  Figure  64,  the  Final  Maintenance  Group 
Balanced  Scorecard,  and  serve  to  answer  the  research  question: 
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What  metrics  should  be  used  in  the  Maintenance  Group  Balanced  Scorecard 


framework  to  meet  the  needs  of  the  CAF  Maintenance  Community? 


Metrics 

Recommendation 

Abort  Rate  (Air/Ground/Total) 

Incorporate 

Aircraft  Availability 

Incorporate 

Aircraft  Inventory  (Backup  (BAI)  and  Primary  (PAI)) 

Eliminate 

Aircraft  Possessed  (Average,  Primary  Possessed  Rate) 

Eliminate 

Average  Sortie  duration  (Actual/Programmed) 

Incorporate 

Break  Rate  (C-3  break  rate) 

Incorporate 

Cannibalization  Rate 

Re-evaluate 

Career  Development  Course  Success  Rate  (Pass  Rate)(Enrolled/Completed) 

Incorporate 

Delayed  Discrepancy  Rate  (Total,  AWP/AWM,  Avg  per  aircraft) 

Incorporate 

Deviation  Rate  (Mx/Ops/ATC/Wx/HHQ/SYM/OTH/SUP/UTE, 
Chargeable/Non-Chargeable) 

Incorporate 

Fix  Rate  (4/8/12-hr) 

Eliminate 

Flying  Schedule  Effectiveness  rate  (FSE) 

Incorporate 

Maintenance  Scheduling  Effectiveness  Rate  (MSE) 

Incorporate 

Manning  Rate 

Incorporate 

Mission  Capable  Rate  (MC,  FMC,  PMC,  NMC,  TNMC,  M/S/B) 

Incorporate 

Mission-Impaired  Capability  Awaiting  Parts  (MICAP) 

Re-evaluate 

Phase/Isochronal  Inspection  Rate  (TDl/Phase  Flow  Average/Phase 

Inspection  Flow/lSO) 

Incorporate 

QA  Pass  Rates  (KTl/PE/RIL/QVI/SI) 

Incorporate 

QATDV/UCR/DSV  Rates 

Incorporate 

Repair  cycle  Time  (Avg  days/Processing  Rate)  (Pre-Mx  Time,  Post-Mx  Time, 
Repair  Time)  (Base  Repair  Cycle  Time)  (Overall  Base  Repair  Cycle  Processing 
Time) 

Incorporate 

Repeat-Recur  Rate  (Repeat  rate.  Recur  Rate) 

Incorporate 

Sorties  Scheduled  (Adjusted/Total) 

Incorporate 

Spare  engine  Status 

Incorporate 

Supply  Effectiveness  Rate  (Issue/Stockage/Benchstock) 

Re-evaluate 

Training  Rate  (No-Shows/Overdues/Training  Effectiveness  Rate) 

Incorporate 

Upgrade  Training  rate  (by  skill  level)  (school  backlogs) 

Incorporate 

UTE  Rate  (Actual/Programmed)  (Sortie/Hourly) 

Incorporate 

Workcenter  Utilization  Rate 

Re-evaluate 

Job  Standard  Fix  Rate 

Create 

Equipment  Maintenance  Scheduling  Effectiveness  Rate 

Create 

Equipment  Effectiveness/Break  Rate 

Create 

Equipment  Fix  Rate 

Create 

Equipment  MICAP  Rate 

Create 

Maintainer  Availability 

Create 

Maintainer  Capability  Rate 

Create 

Equipment  MC/ln-Commission  Rate 

Create 

Equipment  Availability 

Create 

Mission  Effectiveness  Rate 

Create 

Operations  Scheduling  Effectiveness  Rate 

Create 

RAP  Milestones  Rate 

Create 

Maintainer  Experience  Level 

Create 

Maintainer  Skill  Level 

Create 

Overtime  Rate 

Create 

Weekend  Duty  Rate 

Create 

Re-enlistment  Rate 

Create 

Figure  63:  Maintenance  Group  Balanced  Scorecard  Metrics  Recommendations 
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interna 

Process  Perspective 

Aircraft  Mx  Scheduling  and 

Execution 

Equipment  Mx  Scheduling  and 

Execution 

Mx  Training 
Scheduling 
and  Execution 

Scheduled  Mx 

Unscheduled 

Mx 

Scheduled  Mx 

Unscheduled 

Mx 

Maintenance 
Scheduling 
Effectiveness  Rate 

Break  Rate 

Equipment 
Maintenance 
Scheduling 
Effectiveness  Rate 

Equipment 

Effectiveness/ 

Break  Rate 

Career 

Development 

Course  Success 

Rate 

Delayed 

Discrepancy  Rate 

Repeat-Recur  Rate 

Fix  Rate 

Training  Rate 

Phase/lsochronal 
Inspection  Rate 

JST  Fix  Rate 

MICAP  Rate 

Upgrade  Training 

Rate 

Customer  Perspective 


Flying  Scheduling  and 
Execution 


Flying  Schedule 
Effectiveness  Rate 


Sorties  Scheduled 


Abort  Rate 


Deviation  Rate 


Average  Sortie 
duration 


Aircrew 

Mission/ 

Training 

Scheduling 


Mission 

Effectiveness  Rate 


Operations 
Scheduling 
Effectiveness  Rate 


RAP  Milestones 
Rate 


Maintenance 
Group  Strategic 
Objective: 
Maintain  aircraft, 
equipment  and 
personnel  in  a  safe, 
serviceable  and 
ready  condition  to 
facilitate  mission¬ 
readiness  of  the 
flying  wing. 


Readiness  Perspective 


Maintainer 

readiness 


Manning  Rate 


Maintainer 

Availability 


Maintainer 

Capability 


Aircraft 

readiness 


Aircraft  Availability 


Mission  Capable 
Rate 


Equipment 

readiness 


Spare  Engine 
Status 


Equipment  MC 
Rate/ln-commision 
Rate 


Equipment 
Availability  Rate 


Workforce  Quality  Perspective 

Mx  Quality, 
Safety  and 
Discipline 

Maintainer 

Skill  and 

Experience 

Maintainer 

Retention 

QA  Pass  Rates 

Maintainer  Skill 

level 

Re-enlistment 

Rate 

QA  TDV/UCR/DSV 

Rates 

Maintainer 
Experience  level 

Weekend  Duty 

Rate 

Overtime  Rate 

Figure  64:  Final  Maintenance  Group  Balanced  Scorecard 


These  recommendations  for  refinement,  addition,  or  reassessment  of  metrics  were 
compared  to  the  results  of  related  metrics  research  by  Capt  Brian  Waller  and  Capt  Emily 
Harris,  and  Nachtman,  et  al,  whose  work  was  mentioned  in  the  Literature  Review. 
Although  the  methodologies  used  in  each  of  these  pieces  of  research  differed,  and  the 
scopes  of  each  piece  of  literature  varied,  there  were  several  similarities  in  metrics 
proposed  for  use  or  recommended  for  incorporation,  reevaluation  or  elimination.  The 
results  of  this  comparison  are  listed  in  Appendix  K. 
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Other  Recommendations 


Recommend  the  standardization  of  maintenance  metrics  vocabulary  between 
publications  where  they  are  addressed,  and  expand  the  metrics  study  publication,  ACCI 
21-1 18  to  specifically  list  recommended  metrics  to  be  studied  at  the  Group  Level  from 
the  Maintenance  Group  Balanced  Scorecard  framework  presented,  including 
recommended  new  metrics,  and  recommended  refined  metrics. 

Recommend  the  centralization  of  Maintenance  Group  Analysis  shops  at  the  wing 
level  to  perform  analysis  functions  for  both  the  Operations  Group  and  the  Maintenance 
Group  as  the  Wing  Analysis  function  to  facilitate  a  holistic  view  of  the  performance  of 
the  entire  flying  wing,  instead  of  purely  the  performance  of  the  Maintenance  Groups. 

Recommendations  for  Future  Research 

First,  a  Delphi  Study  could  be  performed  using  the  results  of  this  study  to  obtain 
follow-up  inputs  on  the  final  proposed  Maintenance  Group  Balanced  Scorecard, 
including  formulas  for  the  metrics,  and  incorporating  a  Supplier  Perspective,  as 
mentioned  in  recommendations. 

A  follow-on  study  could  also  address  issues  of  data  integration  with  current 
maintenance  data  system  capabilities,  as  maintenance  infonnation  systems  such  as 
Integrated  Maintenance  Data  System  or  GO-8 1  may  not  currently  have  the  capability  to 
capture  required  data  to  implement  proposed  metrics. 

The  need  for  a  Supplier  Performance  Perspective  was  a  revelation  revealed  late  in 
the  research  process  and  warrants  further  consideration  and  research  beyond  the  scope  of 
this  thesis.  The  researcher  strongly  recommends  further  research  to  validate  including  a 
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Supplier  Performance  Perspective  on  the  Maintenance  Group’s  Balanced  Scorecard  as 
Supplier  Perfonnance  measures  would  be  especially  relevant  to  5th  generation  aircraft 
under  the  influence  of  the  corporate  contributors,  vendors  and  manufacturers.  Supplier 
Perfonnance  measures  are  an  absolute  necessity  to  limit  sustainment  vulnerabilities  as 
programs  such  as  the  F-22,  F-35  and  KC-46  mature. 

To  clearly  understand  the  Maintenance  Group’s  value  proposition  in  dealing  the 
Aircrew  as  the  customer,  a  follow-on  study  could  expand  survey  to  include  pilots  in  the 
Operations  Groups  up  to  the  MAJCOM  and  HAF-level  A3. 

Finally,  since  this  research  was  focused  on  CAF  metrics,  future  research  could 
expand  this  study  to  Mobility  Air  Forces  (MAF)  and  their  unique  set  of  aircraft 
maintenance  and  operations  metrics. 

Conclusions 

Since  the  beginning  of  Aircraft  Maintenance,  leaders  have  relied  upon 
perfonnance  metrics  as  tools  to  help  guide  decisions,  improve  processes  and  maximize 
perfonnance.  Maintainers  and  their  stakeholders  continue  to  need  to  plainly  see  and 
understand  the  link  between  the  processes  of  a  Maintenance  Group,  the  customers  they 
provide  for,  the  maintainers  that  enable  it,  and  the  bottom-line  readiness  of  all  entities 
involved.  The  Balanced  Scorecard  approach  to  looking  at  maintenance  metrics  first  helps 
to  focus  the  Maintenance  Group’s  analysis  around  their  strategic  objective  and  helps 
leaders  to  understand  who  their  customer  is,  what  processes  are  critical  to  achieving  their 
strategic  objectives  and  customer’s  needs,  and  gives  a  bottom-line  view  of  readiness  of 
the  entities  of  the  Maintenance  Group.  Finally,  the  Maintenance  Group  Balanced 
Scorecard  framework  helps  leaders  and  decision-makers  to  view  the  health  of  the 
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enablers  of  the  entire  operation— the  maintainers— and  understand  factors  that  contribute 
to  their  job  satisfaction,  the  quality  of  their  work,  and  their  skills  and  experience  in  their 
work. 

An  organization  as  complex  and  dynamic  as  the  Maintenance  Group  demands  a 
metrics  framework  that  is  comprehensive  enough  to  understand  all  aspects  of  the 
perfonnance  of  the  organization,  but  structured  enough  to  directly  see  how  positive  or 
negative  dynamics  of  one  process  may  be  affecting  another.  Metrics  have  helped  to 
guide  Aircraft  Maintenance  through  decades  of  challenge  and  change;  with  an  aging 
fleet,  budget  constraints,  and  slashed  manning  levels  as  the  backdrop  for  current  and 
future  operations,  there  has  never  been  a  more  critical  time  to  implement  a  clear,  holistic 
metrics  framework  that  meets  the  needs  of  the  leaders  and  decision  makers  who  are 
navigating  through  the  turbulent  times  ahead.. 
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Performance 
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Results 

Performance 
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Service 

Businesses 
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the  balanced 

scorecard—a 
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assumptions 

Metrics  and 
performance 

measurement  in 

operations 
management: 
dealing  with  the 

metrics  maze 
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performance 

measurement 

systems:  a 

literature  review 

An  exploratory 
study  of 
performance 

measurement 

systems  and 
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with 

performance 

results 

Logistics 
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Measurement 

and  Customer 

Success 
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Performance 

Measurement 

Manifesto 

Performance 

measurement 

system  design: 
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,  Douglas  M. 
Stewart,  Morgan 
Swink 

Bourne,  Mike  and 
Andy  Neely 

Evans,  James  R. 
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Cooper 
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Neely,  Andy  and 
John  Mills,  Ken 
Platts,  Huw 
Richards,  Mike 
Gregory, 

Mike  Bourne, 

Mike  Kennerley 
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Appendix  B:  Exhaustive  List  of  Metrics 


AFTTP  3- 

3. Aircraft 

Maintenance 

AFI  11-102 

AFI  36-2232 

ACCI  21-118 

AFI  21-103 

AFLMA  "Blue 

Book" 

AFI  21-165 

AFI  21-101 

Metric  Name 

Tactics, 

Techniques 

and 

Procedures 

Flying  Flour 
Program 
Management 

Maintenance 

Training 

Logistics 

Maintenance 

Performance 

Indicator 

Reporting 

Procedures 

Equipment 

Inventory, 

Status  and 

Utilization 

Reporting 

Aircraft 
Flying  and 
Maintenance 
Scheduling 
Procedures 

Aircraft  and 

Maintenance 

Equipment 

Management 

Abort  Rate  (Air/Ground/Total) 

Aircraft  Availability 

X 

X 

X 

Aircraft  Inventory  (Backup  (BAI) 
and  Primary  (PAI)) 

Aircraft  Possessed  (Average, 
Primary  Possessed  Rate) 

Average  Sortie  duration 
(Actual/Programmed) 

x 

x 

x 

x 

Break  Rate  (C-3  break  rate) 

X 

X 

x 

Cannibalization  Rate 

X 

X 

X 

Cannot  Duplicate  Rate  (CND) 

X 

Career  Development  Course 
Success  Rate  (Pass 
Rate)(Enrolled/Completed) 

Delayed  Discrepancy  Rate  (Total, 
AWP/AWM,  Avg  per  aircraft) 

Depot  Rate 

X 

Deviation  Rate 

( Mx/Ops/ ATC/Wx/H  HQ/S  YM/O 
TH/SUP/UTE,  Chargeable/Non- 
Chargeable) 

EW  Pod  MC/AWP  Rate 

X 

Fix  Rate  (4/8/12-hr) 

X 

X 

X 

Flying  Schedule  Effectiveness 
rate  (FSE) 

Fully  Mission-Capable  Rate 
(FMC) 

Functional  Check  Flight  (FCF) 
Release  Rate 

Hangar  Queens  (Avg,  Avg  Cat 
1/2/3) 

x 

Isochronal/Phase  Inspection 

Rate  (TDI/Phase  Flow 
Average/Phase  Inspection 
Flow/ISO) 

LANTIRN  POD  MC  Rate 

X 

LANTIRN  Test  Station  MC  Status 

Maintenance  Man-Hours  per 
Flying  Hour 

Maintenance  Scheduling 
Effectiveness  Rate  (MSE) 

x 

x 

x 

Manning  Rate 

X 

X 

Mission-Capable  Rate  (MC) 

X 

X 

X 

X 

Mission-Impaired  Capability 
Awaiting  Parts  (MICAP) 

MMCL  Backlog 

Monthly  Job  Data 
Documentation  (JDD)  Error  Rate 
(Before/ After  Corrections) 

MTF  and  FTD  Demand  Response 
and  Utilization  Rate 

Not  Mission  Capable  (NMC) 
(Supply/Mx/Both)  Rate 

x 

x 

x 

Partially  mission  capable  rate 
(PMC)  (Supply/Mx/Both) 

Percent  of  Munitions  (2WO) 
task  qualified  >70% 

Personnel  Availability  rate  (PA) 

X 

QA  Pass  Rates 
( KTL/PE/RI L/QVI/SI ) 

QA  TDV/UCR/DSV  Rates 

X 

X 

Repair  cycle  Time  (Avg 
days/Processing  Rate)  (Pre-Mx 
Time,  Post-MxTime,  Repair 
Time)  (Base  Repair  Cyde  Time) 
(Overall  Base  Repair  Cycle 
Processing  Time) 

Repeat-Recur  Rate  (Repeat 
rate.  Recur  Rate) 

x 

x 

Sorties  Scheduled 
(Adjusted /Total) 

x 

Spare  engine  Status 

X 

X 

Special  Experience  Identifier 

Supply  Effectiveness  Rate 
(Issue/Stockage/Benchstock) 

Total  Not  Mission-Capable  rate 
(TNMC)  (Supply/Mx/Both) 

x 

Training  Rate  (No- 

Sh  o  ws/Ove  rd  u  es/Tra  i  n  i  ng 

Effectiveness  Rate) 

Unit  Possessed  Not  Reported 
(UPNR) 

x 

Upgrade  Training  rate  (by  skill 
level)  (school  backlogs) 

x 

x 

UTE  Rate  (Actual/Programmed) 
(Sortie/Hourly) 

Weapons  Release 

Reliability/Gun  Fire-out  Rates 

Workcenter  Utilization  Rate 

X 

X 

127 


Appendix  C:  Consolidated  List  of  Metrics 


AFTTP  3- 

3.  Aircraft 

Maintenance 

AFI 11-102 

AFI  36-2232 

ACCI  21-118 

AFI  21-103 

AFLMA 

"Blue 

Book" 

AFI  21-165 

AFI  21-101 

Metric  Name 

Tactics, 

Techniques 

and 

Procedures 

Flying  Hour 
Program 
Management 

Maintenance 

Training 

Logistics 

Maintenance 

Performance 

Indicator 

Reporting 

Procedures 

Equipment 

Inventory, 

Status  and 

Utilization 

Reporting 

Aircraft 
Flying  and 
Maintenance 

Scheduling 

Procedures 

Aircraft  and 

Maintenance 

Equipment 

Management 

Abort  Rate  (Air/Ground/Total) 

X 

X 

X 

Aircraft  Availability 

X 

X 

X 

Aircraft  Inventory  (Backup  (BAI) 
and  Primary  (PAI)) 

X 

X 

Aircraft  Possessed  (Average, 
Primary  Possessed  Rate) 

X 

X 

Average  Sortie  duration 
(Actual/Programmed) 

X 

X 

X 

X 

Break  Rate  (C-3  break  rate) 

X 

X 

X 

Cannibalization  Rate 

X 

X 

X 

Career  Development  Course 
Success  Rate  (Pass 
Rate)(Enrolled/Completed) 

X 

X 

X 

Delayed  Discrepancy  Rate  (Total, 
AWP/AWM,  Avg  per  aircraft) 

X 

X 

X 

Deviation  Rate 

(Mx/Ops/ ATC/Wx/HHQ/SYM/O 
TH/SUP/UTE,  Chargeable/Non- 
Chargeable) 

X 

X 

X 

Fix  Rate  (4/8/12-hr) 

X 

X 

X 

Flying  Schedule  Effectiveness 
rate  (FSE) 

X 

X 

X 

X 

Fully  Mission-Capable  Rate 
(FMC) 

X 

X 

Isochronal/Phase  Inspection 

Rate  (TDI/Phase  Flow 
Average/Phase  Inspection 
Flow/ISO) 

X 

X 

Maintenance  Scheduling 
Effectiveness  Rate  (MSE) 

X 

X 

X 

X 

Manning  Rate 

X 

X 

Mission-Capable  Rate  (MC) 

X 

X 

X 

X 

Mission-Impaired  Capability 
Awaiting  Parts  (MICAP) 

X 

X 

Not  Mission  Capable  (NMC) 
(Supply/Mx/Both)  Rate 

X 

X 

X 

Partially  mission  capable  rate 
(PMC)  (Supply/Mx/Both) 

X 

X 

QA  Pass  Rates 
(KTL/PE/RIL/QVI/SI) 

X 

X 

QA  TDV/UCR/DSV  Rates 

X 

X 

Repair  cycle  Time  (Avg 
days/Processing  Rate)  (Pre-Mx 
Time,  Post-Mx  Time,  Repair 
Time)  (Base  Repair  Cyde  Time) 
(Overall  Base  Repair  Cycle 
Processing  Time) 

X 

X 

Repeat-Recur  Rate  (Repeat 
rate,  Recur  Rate) 

X 

X 

Sorties  Scheduled 
(Adjusted/Total) 

X 

X 

Spare  engine  Status 

X 

X 

Supply  Effectiveness  Rate 
(Issue/Stockage/Benchstock) 

X 

X 

X 

Total  Not  Mission-Capable  rate 
(TNMC)  (Supply/Mx/Both) 

X 

X 

X 

X 

Training  Rate  (No- 

Sho  ws/Ove  r  d  u  es/Tr  a  i  n  i  ng 

Effectiveness  Rate) 

X 

X 

X 

Upgrade  Training  rate  (by  skill 
level)  (school  backlogs) 

X 

X 

X 

UTE  Rate  (Actual/Programmed) 
(Sortie/Hourly) 

X 

X 

X 

X 

Workcenter  Utilization  Rate 

X 

X 
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Appendix  D:  Mandated  Meeting  Frequency  and  Content 


Meeting: 

Level  Held: 

Required 

Frequency: 

Required  Participants 

Recommended 

participants 

Purpose  of  meeting 

Required  agenda 

Recommended  Agenda 

Metrics  discussed? 

1 

tci  Meeting 

N 

2 

TCTO  review  meeting 

Monthly 

PS&D  (Chair) 

supply  reconc&Mien  supply  status, 
schedulng  factors,  current  I CIO  status 
and  arilMpulrd  protArmv  for  all  active 

TCTOs 

N 

QA 

3 

Pre/Post  Dock 

Meeting 

Aircraft  Section  Technicians 

N 

Specials!  section  NCOK 

Units  pubtshmOl  to 
determinc/standardirc 

PS&D 

4 

MSEP 

Group 

Squadron  CC 

Review  MSEP  data 

Y 

Squadron  MOO/Suot 

WWM 

duet  inspector 

Senior  Analysis 

MSEP  monthly 
summary  (report) 

Monthly 

The  MSEP  summary  wll  avlude  visual 
information.  graphs.  narratares.  cualfv 
trends  Idemifed  through  inspectors  and 
evahiatiors,  ctscussion  of  common 
prohiem  areas  and  descriptions  of 
successful  programs  or  initiatives 

r 

uxnpfcance  with  and  currency  of  lOs 
and  diteo trees  to  delude  unit. 

8.10. 10.7  Aircraft  and  equipment  forms 
docureotaticav 

81019.3  CompSance  and  Management 
of  safety,  Lnwr  on  mental.  Housekeeping, 
and  fOD  Programs. 

8.1019.4  Training  Prograrri. 

8.10. 19.S  Key  Task  List  IKTl). 

8  1019.6  Routine  Inspection  1st  (RR). 
8.1019.7.  Msi retons  Program. 

8  10  19.8  Hagfr-missed  raided  items 

8.1019.9  Narrative  Repot  The 
mrotNy  narrat we  recoct  must  contain 

an  analysts  of  the  MSt**  results,  a 
summary  of  significant  CAT  I  and  II 
cSscrepanties.  technical  inspections  and 
recommendations  for  improvement. 

Prise  to  preparing  the  narrative  report, 
QA  nos:  conduct  a  study  of  trends 

8  10 19  10  Trend  Analysis 

5 

MSEP 

Wirg 

Quarterly 

Wing  CC  (Chair) 

Qua  Sty  issues 

Y 

actionable  fccdoacs  lor  unit  leadership 
concern  ng  over  a*  health  of  nut 

stay  abreast  of  Mx  Issuers 

C 

IREP 

Mil 

Quarterly 

Wing/CV  (Chart' 

IRS 

rcroaso  overall  base 
self  suffriency  far 

repay  and  reduce  the 

over*  cost  of 

operations 

key  elements  of  asset  maragerrent  and 
costs  associated  with  each  of  the 

maintenance  stock  fund  divisions 

Y 

MXG/CC 

IRS.FSC. 

itiv  " 

frcm  nalntenance 

liivts,  O&M 

resource 

advisors, 

maintenance 

analysis.  AFRtP  (  f 
.  l>jUallt),  QA 

and  others 

Asset  PioHr/fop Projected  MICAP 
Situetons 

MSG/C  C 

Test  Station  equipment  Piofil* 

Wing  iet-bufrtciencv  Imoatves 

High  Cost  Maintenance 

Top  CAM!  Items 

Unit  Air  u  aft  Engine  Status  Review 

Repair  Cycle  Bottlenecks 

AWP  Summary 

Repair  Cycle  Throughput 

Part  Store  Issue  Iffecthreness 

Discuss  product  improvement  initiatives 
and  maintenance  related  to  Innovation 
Development  through  Employee 
Awareness  [IDEA)  subnr  sswtis 

7 

Group 

Quarterly 

critical  support  equipment  status 

Y 

129 


R 

PIP  R&M  Meeting 

Group 

MXG/CC  (Cnabl 

y 

9 

Wing  Standup 

wue 

WmgComnwnder  (Chav) 

llevenv  of  previous  day  activities 

r 

Review  of  ojtrent  day  acikeiet 

Identify  ard  resove  nsi.es  wnh 
ejecuting  established  mrinlenjnce 
scheduto 

Cat  2  and  3  CANN  Aircraft  M*  and  Supply 
Status 

10 

Scheduling  Meeting 

w«* 

Wm*  Commander  (Chad) 

Stay  abreast  of  Mx  Issues 

r 

deal:  acad  srhedutor 

oust  nut  past  auomifeJimeits,  status  of 
ttym*  gewds,  proMems  encountered  and 
deraded  needs  for  the  netf  month.  PSRO 
outlies  projected  maintenance 
copstVrty  and  airtralt/equiprrent 

maldAy  If conflicts arise between 

opsratsonal  requirements  and 
maintenance  capabiStv.  present 
afcet  tulf/vs  and  1  ir stations.  MXG/CC, 
OG/tl  and  WG/CC  iNode  «h at  portion 
of  the  mission  to  support  and  to  wtwr 
deg 

AVOO 

11 

Munitions  Scheduling 
and  Training  Meeting 

Grcup-Mun 
kion*  Fight 

MonthM’l 

12 

Meeting  to  establish 
prionties 

Group 

beginning  of 

Mudse  and 
continginues 

? 

13 

Mx  Production  and 
Scheduling  Meeting 

Group 

(Mr 

MXG/CD(Ch»r) 

Verily  aircraft  and 
ecjuipinenl  utfcretiim 
requirements  lor  next 
day 

ae-cr  aft  status 

r 

vxr./r  D  designates 
mandatory  atlondtas 

scheduled 

maintenance 

requirments  (nr  oral 
day 

MIUW  ard  Kopalr  lydo  status 

Squadron  MOO/Supt 

esuHshwork 

priorities 

A(  IMT 1407  actions 

MOC  Senior  Cociufcnar.or 

coordinate  schedule 

clunget 

cut  tent  day  ft*i!«  twcutiou 

(M  Section 

frnuiv  critical 
equipiror*,  faciir.Ks, 
•ndmatenH 

resources  are 

allocated  and  cstaban 
overal  priorities 

current  day  ms  execution 

AVOO 

remain  log  portion  of  current  day 
schedule 

eesiew  previous  day  tiyvig  derialwns 

remew  previous  day  mx  schecu  e 

deviations 

supply  miraps 

pnorltortg  aircraft  rcounrg  sharod 

resources 

•rswsvSI.TCI.  TaOs 

itsww  0FT/CFT  suliv  Jufcrs 

defy  resofcition  of  configuration 
manage irent  notices  u’Juing  screen  UWj 
or  applicable  MIS  screen  discrepancies 

14 

weekly 

revew  nod  week's  tlym*  and  ntx 
schedule 

Y 

rcsWsv  overdue  speoal  rapsctiois  and 

TC  s 

tesww  TCTOs  that  wN  ground  withes  JO 
dm 

DFt/CFT  schedule  requirements 

remove  previous  week  s  Carnations  to 
flying  schedule 

review  previous  week  t  donations  to  mx 
schedule 

15 

Status  of  Training 

Squadron 

monthly 

Squadron  CC 

Insure  upgrade 
train  in*  end 

quakttcaoon  prog-amt 
e-nohasu-e  quality  and 
*r*  not  primeiily 

tocused  on  meconit 
minimum  upgrade 
time  frames 

V 

10 

Status  of  Training 
Briefing 

Group 

Group  CC 

r 

17 

Status  of  Training 
Briefing 

wng 

Quartette 

wmgee 

Y 

1* 

AMU  Maintenance 
Production  Meeting 

AMU/F%M 

i- *'v 

AMU  OlC/Vjpr 

Y 

19 

DITIVleeting 

Lead  Command 

Mirawwi 

Y 

20 

Scheduling  Meeting 

Group 

Quarterly  (Nil  u 
days  before  the 
next  quarter) 

MXG/CC  OG/CC  (cue  irl 

unit's  queiterly  pUn  aid  includes 
operational  requirements,  support 
capsWity  and  any  ddficulties  eweeted 

r 

dednated  srlwduMr 

AVOO 

21 

W»«v 

dedicated  scheduler 

es a'uate  ll»e  p*sl  week's 
accomefsh roams  (to  Helude  Ifylng  ate 
MSI  I  and  negotiate/aporove 
refinements  to  the  c  erring  wert‘% 
schedule 

AVOO 

130 


ii 

Wing  Weapons 
Meeting 

wir* 

WWM(Otatr) 

Weapons  A- 1  IS 

d<scos*  and  resolve 
any  wing  weapons 

issues,  concerns  or 

problems 

WS 

wing  safety 

quakry  assu'ance 

muntioris  fight 

armament  fhRht 

section* 

n 

POO  Prevention 

Committee 

Mjnt'ily  when 

units  ercccd 

standard 

Wi  ng/CV 

Identify  neyjat  ve 
trends  and  develop 
acton  ptons  to  resche 

them 

Total  number  of  airframe,  engine,  and 
are  FOD  incidents  dunng  tfe 
reporting  period.  Indicate  quantity 
and  cause.  Current  status  cf  all  other 
pending  incident*  tuM  be  dkcussed 

r 

Quarterly  if  fOO 
rates  ess  than 

standard 

Group  Commanders 

Meslunna /vacuum  sweeper  status 

Director* 

Revew  and  refinement  of  the  easting 
TOO  prevention  program 

Commander*  of  erst*  with  Ms 

personnel 

Hew  directrvas/acticc**  estaUitheo  to 
mmrmijeFOD. 

Safety 

sums  and  condition  of  enffne  run  up 
screens  as  appliceljtf . 

a 

Results  of  X-rays  for  TOs  during 
ergne  bay  inspection*,  acceptance 
mpectioo*.  and  phase  mpertion*. 

MamtehcTKe  trends  shotid  be 

discussed  when  an  increase  in  FO  is 
discovered  during  these  X  rays. 

Airfielr  Manager 

Identification  of  potential  FOD 

sources 

Security  Forces 

Lost  tooh/korrs 

1  enant  Unit  IOD  committees 

Increased  potential  for  FOD  within  the 
hen  dO-eOdays 

Dropped  objects,  eay  particular 
attention  to  those  that  result  in 

downstream  FOO. 

Breakdown  of  FOD 
nscectonsMsessments 

Cockpit  FO  i  readers* 

Commanders  comments. 
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Appendix  E:  Maintenance  Metrics  Survey 


Purpose  of  Survey 


PURPOSE: 

The  purpose  of  this  survey  is  to  collect  and  analyze  the  opinions  of  maintenance  experts  regarding  maintenance  metrics 
used  in  the  CAP  aircraft  maintenance  community-  The  results  of  this  survey  will  be  used  to: 

1)  Explore  the  utility  of  a  metrics  framework  called  "balanced  scorecard"  for  use  in  a  Maintenance  Group 

2)  Explore  the  optimal  frequency  of  metrics  analysis  at  each  maintenance  management  level. 

3)  Evaluate  the  "goodness"  individual  maintenance  metrics. 

4)  Aid  in  determining  whether  the  current  metrics  set  sufficiently  meets  the  needs  of  decision  makers  who  use  them. 

5)  Recommend  changes  to  the  existing  metrics  framework  to  Headquarters  Air  Force  Director  of  Logistics  (AF/A4L). 
CONFIDENTIALITY: 

All  answers  will  be  kept  strictly  confidential.  In  no  way  will  the  information  you  provide  be  used  to  determine  who  you  are. 
The  demographic  information  is  valuable  to  this  research  and  will  only  be  used  in  analysis  of  the  results.  No  one  other 
than  the  researcher  will  see  your  responses.  Group  trends  and  statistical  findings  may  be  published  and  briefed  to 
leadership  personnel  as  part  of  this  research. 

PARTICIPATION: 

Participation  is  strictly  voluntary.  You  are  not  required  to  participate  in  this  survey.  This  survey  should  take  approximately 
20  to  30  minutes  to  complete.  You  may  exit  this  survey  and  return  to  it  at  any  time  if  you  do  not  finish  on  your  first 
attempt. 

INSTRUCTIONS: 

•  Please  base  your  answers  on  your  own  thoughts  and  experiences 

•  Please  make  your  answers  clear  and  concise  when  asked  to  answer  in  a  response  or  when  providing  comments 

•  Be  sure  to  select  the  correct  option  button  when  asked 

CONTACT: 

If  you  have  questions  about  this  survey  please  contact  adrienne.stahl@afit.edu. 
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■oooooo  -ooooooooooooooo 


Maintenance  Experience  demographics 


4  What  is  your  rank? 


/hich  most  closely  describes  your  current  MAJCOM  (or  equivalent)? 


Direct  Reporting  Unit  (DRU) 


Forward  Operating  Agency  (FOA) 


Other  (please  specify) 
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3.  What  is  the  management  level  of  your  current  position? 

Flight  Commander 

o  AMU  OIC 

EMS/CMS/MXS/Muns/MOS  Operations  Officer 
o  AMXS  Operations  Officer 
Squadron  Commander 
Deputy  or  Group  Commander 
Q  MAJCOM 
Depot 

o  Other  (please  specify) 


4.  How  many  years  of  experience  do  you  have  in  Aircraft  Maintenance? 

I - 1 


5.  How  many  years  of  experience  do  you  have  in  a  Combat  Air  Forces  (CAF)  flying  wing? 

I - 1 


6.  What  levels  of  maintenance  have  you  managed?  Please  select  all  that  apply. 

□  Flight  Commander 

□  AMU  OIC 

^  EMS/CMS/MXS/Muns/MOS  Operations  Officer 

|  AMXS  Operations  Officer/Squadron  Maintenance  Officer  (SMO) 

□  Squadron  Commander 

|  Deputy  or  Group  Commander 
MAJCOM 
|  Depot 

□  Other  (please  specify) 


7.  How  much  experience  do  you  have  analyzing  or  reporting  maintenance  metrics? 

I - 1 
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Balanced  Scorecard  Overview 


The  purpose  of  this  portion  of  the  survey  is  to  assess  the  utility  of  the  balanced  scorecard  concept  for  use  in  the 
Maintenance  Group. 

The  balanced  scorecard  is  a  metrics  framework  that  suggests  a  "balanced"  focus  on  metrics  represented  in  four 
perspectives-  The  four  perspectives  are  defined  as: 

1)  Customer  Perspective:  Entity  the  Maintenance  Group  provides  for 

2)  Readiness  Perspective:  Bottom-line  mission  or  combat  capability 

3)  Processes  Perspective:  Actions  required  to  meet  customer  needs 

4)  Workforce  Quality  Perspective:  Ability/skill  to  meet  standards 

The  survey  questions  will  assess  which  existing  metrics  could  fall  under  each  perspective,  and  identify  where  shortfalls  in 
metrics  representing  each  perspective  may  exist. 
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Assessment  of  a  Maintenance  Group's  Strategic  Objectives 


Most  CAF  Maintenance  Groups  have  vision  and  mission  statements  describing  their  goals  and  objectives;  however,  these 
vision  and  mission  statements  can  vary  from  wing  to  wing 

The  purpose  of  this  section  is  to  determine  the  over-arching  strategic  objectives  of  a  typical  CAF  Maintenance  Group 

8.  What  is  your  assessment  of  the  following  statement? 

The  primary  strategic  objective  of  a  CAF  Maintenance  Group  is  to  facilitate  and  ensure 
combat/mission-readiness  of  the  flying  wing. 


Strongly  Disagree 

Disagree 

Neutral 

Agree 

Strongly  Agree 

o 

O 

o 

o 

o 
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Balanced  Scorecard  Perspective  #1:  Maintainers  as  Customers 


The  purpose  of  this  section  is  to  identify  the  customers)  for  use  in  the  Maintenance  Group  balanced  scorecard. 

*9.  Do  you  believe  the  Maintenance  Group  should  view  Maintainers  as  customers,  in 
terms  of  providing  them  support,  training  or  services? 

O Yes 

o° 

o  I  don't  know 

Comments: 
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ooo 


Balanced  Scorecard  Perspective  #1:  Ops/ Aircrew  as  Customer 


*10.  Do  you  believe  the  Maintenance  Group  should  view  Aircrew  as  customers,  in  terms 
of  providing  them  support,  training  or  services? 

Yes 
No 

I  don't  know 

Comments: 
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ooo 


Balanced  Scorecard  Perspective  #1:  Equipment  as  Customer 


*11.  Do  you  believe  the  Maintenance  Group  should  view  Equipment  as  a  customer,  in 
terms  of  providing  support,  training  or  services? 

Yes 
No 

I  don't  know 

Comments: 
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ooo 


Balanced  Scorecard  Perspective  #1:  Aircraft  as  Customer 


*12.  Do  you  believe  the  Maintenance  Group  should  view  Aircraft  as  a  customer,  in  terms 
of  providing  support,  training  or  services? 

Yes 
No 

I  don't  know 

Comments: 
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Balanced  Scorecard  Perspective  #2:  Processes  for  Maintainers  as  Customer 


The  purpose  of  this  section  is  to  identify  the  critical  processes  in  the  Maintenance  Group  that  ensure  the  satisfaction  of 
their  customers. 

1 3.  Please  rate  the  importance  of  each  of  the  following  processes  in  providing  for 
Maintainers  as  the  customer. 


Example:  The _ process  is  critical  in  providing  for  Maintainers  as  the  customer. 


Of  little 

Moderately 

critical 

unimportant 

importance 

important 

mp  an 

Aircraft  flying  scheduling 

o 

o 

o 

o 

o 

Aircraft  flying  execution 

o 

o 

o 

o 

o 

Aircraft  maintenance  scheduling 

o 

o 

o 

o 

o 

Aircraft  maintenance  execution 

o 

o 

o 

o 

o 

Equipment  maintenance  scheduling 

o 

o 

o 

o 

o 

Equipment  maintenance  execution 

o 

o 

o 

o 

o 

Maintenance  training  scheduling 

o 

o 

o 

o 

o 

Maintenance  training  execution 

o 

o 

o 

o 

o 

Aircrew  Mission/Training  scheduling 

o 

o 

o 

o 

o 

Aircrew  Missions raining  execution 

o 

o 

o 

o 

o 

Comments: 
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Balanced  Scorecard  Perspective  #2:  Processes  for  Aircrew  as  Customers 


The  purpose  of  this  section  is  to  identify  the  critical  processes  in  the  Maintenance  Group  that  ensure  the  satisfaction  of 
their  customers. 

14.  Please  rate  the  importance  of  each  of  the  following  processes  in  providing  for  Aircrew 
as  the  customer. 


Example:  The _ process  is  critical  in  providing  for  Aircrew  as  the  customer. 


Of  little 

Moderately 

critical 

unimportant 

importance 

important 

mp  an 

Aircraft  flying  scheduling 

o 

o 

o 

o 

o 

Aircraft  flying  execution 

o 

o 

o 

o 

o 

Aircraft  maintenance  scheduling 

o 

o 

o 

o 

o 

Aircraft  maintenance  execution 

o 

o 

o 

o 

o 

Equipment  maintenance  scheduling 

o 

o 

o 

o 

o 

Equipment  maintenance  execution 

o 

o 

o 

o 

o 

Maintenance  training  scheduling 

o 

o 

o 

o 

o 

Maintenance  training  execution 

o 

o 

o 

o 

o 

Aircrew  Mission/Training  scheduling 

o 

o 

o 

o 

o 

Aircrew  Missions raining  execution 

o 

o 

o 

o 

o 

Comments: 
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Balanced  Scorecard  Perspective  #2:  Processes  for  Equipment  as  Customer 


The  purpose  of  this  section  is  to  identify  the  critical  processes  in  the  Maintenance  Group  that  ensure  the  satisfaction  of 
their  customers. 

1 5.  Please  rate  the  importance  of  each  of  the  following  processes  in  providing  for 
Equipment  as  the  customer. 


Example:  The _ process  is  critical  in  providing  for  Equipment  as  the  customer. 


Of  little 

Moderately 

critical 

unimportant 

importance 

important 

mp  an 

Aircraft  flying  scheduling 

o 

o 

o 

o 

o 

Aircraft  flying  execution 

o 

o 

o 

o 

o 

Aircraft  maintenance  scheduling 

o 

o 

o 

o 

o 

Aircraft  maintenance  execution 

o 

o 

o 

o 

o 

Equipment  maintenance  scheduling 

o 

o 

o 

o 

o 

Equipment  maintenance  execution 

o 

o 

o 

o 

o 

Maintenance  training  scheduling 

o 

o 

o 

o 

o 

Maintenance  training  execution 

o 

o 

o 

o 

o 

Aircrew  Mission/Training  scheduling 

o 

o 

o 

o 

o 

Aircrew  Missions raining  execution 

o 

o 

o 

o 

o 

Comments: 


143 


Balanced  Scorecard  Perspective  #2:  Processes  for  Aircraft  as  Customer 


The  purpose  of  this  section  is  to  identify  the  critical  processes  in  the  Maintenance  Group  that  ensure  the  satisfaction  of 
their  customers. 

16.  Please  rate  the  importance  of  each  of  the  following  processes  in  providing  for  Aircraft 
as  the  customer. 


Example:  The _ process  is  critical  in  providing  for  Aircraft  as  the  customer. 


Of  little 

Moderately 

critical 

unimportant 

importance 

important 

mp  an 

Aircraft  flying  scheduling 

o 

o 

o 

o 

o 

Aircraft  flying  execution 

o 

o 

o 

o 

o 

Aircraft  maintenance  scheduling 

o 

o 

o 

o 

o 

Aircraft  maintenance  execution 

o 

o 

o 

o 

o 

Equipment  maintenance  scheduling 

o 

o 

o 

o 

o 

Equipment  maintenance  execution 

o 

o 

o 

o 

o 

Maintenance  training  scheduling 

o 

o 

o 

o 

o 

Maintenance  training  execution 

o 

o 

o 

o 

o 

Aircrew  Mission/Training  scheduling 

o 

o 

o 

o 

o 

Aircrew  Missions raining  execution 

o 

o 

o 

o 

o 

Comments: 
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Balanced  Scorecard  Perspective  #2:  Processes 


17.  Which  of  the  following  processes  does  the  Maintenance  Group  have  an  impact  on? 
Please  select  all  that  apply. 

□  Flying  scheduling 
~|  Flying  execution 

|  Aircraft  maintenance  scheduling 
|  Aircraft  maintenance  execution 
|  Equipment  maintenance  scheduling 

□  Equipment  maintenance  execution 
Maintenance  training  scheduling 

□  Maintenance  training  execution 
~~|  Aircrew  Mission/Training  scheduling 

□  Aircrew  Mission/Training  execution 
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18.  Please  select  the  metric(s)  that  depict  the  efficiency/effectiveness  of  each  process 
listed. 


Example:  The _ rate  depicts  the  efficiency/effectiveness  of  the  Flying  Scheduling 

Process. 


Flying  Scheduling  Process 

Flying  Execution  Process 

Abort  Rate  (Air/Ground/Total) 

□ 

□ 

Aircraft  Availability 

□ 

□ 

Aircraft  Inventory  (Backup  (BAI)  and  Primary  (PAI)) 

□ 

□ 

Aircraft  Possessed  (Average,  Primary  Possessed  Rate) 

□ 

□ 

Average  Sortie  duration  (Actual/Programmed) 

□ 

□ 

Break  Rate  (03  break  rate) 

□ 

□ 

Cannibalization  Rate 

□ 

□ 

Career  Development  Course  Success  Rate  (Pass  RateXEnrolled/Completed) 

□ 

□ 

Delayed  Discrepancy  Rate  (Total.  AWP/AWM.  Avg  per  aircraft) 

□ 

□ 

Deviation  Rate  (Mx/Ops/Vto,  etc)(Chargeable/Non-Chargeable) 

□ 

□ 

Fix  Rate  (4/8/12-hour) 

□ 

□ 

Flying  Schedule  Effectiveness  Rate  (FSE) 

□ 

□ 

Maintenance  Scheduling  Effectiveness  Rate  (MSE) 

□ 

□ 

Manning  Rate 

□ 

□ 

Mission  Capable  Rate  (MC.  FMC.  PMC,  NMC,  TNMC,  M/S/B) 

□ 

□ 

Mission-Impaired  Capability  Awaiting  Parts  (MICAP) 

□ 

□ 

Phase/lsochronal  Inspection  Rate  (TDI/Phase  Flow  Average/ISO) 

□ 

□ 

QA  Pass  Rates  (KTL/PE/RIL/QVI/SI) 

□ 

□ 

QA  TDV/UCR/DSV  Rates 

□ 

□ 

Repair  cycle  Time  (Avg  days)  (Pre/Post-Mx  Time.  Repair  Time) 

□ 

□ 

Repeat-Recur  Rate  (Repeat  rate,  Recur  Rate) 

□ 

□ 

Sorties  Scheduled  (Adjusted/Total) 

□ 

□ 

Spare  Engine  Status 

Supply  Effectiveness  Rate  (Issue/Stockage/Benchstock) 

□ 

□ 

Training  Rate  (No-Shows/Overdues/Training  Effectiveness  Rate) 

□ 

□ 

Upgrade  Training  Rate  (by  skill  level)  (school  backlogs) 

□ 

□ 

UTE  Rate  (Actual/Programmed,  Sortie/Hourly) 

□ 

□ 

Workcenter  Utilization  Rate 

□ 

□ 

None  of  the  above 

□ 

□ 
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19.  Please  assess  the  following  statement: 

There  are  adequate  metrics  to  depict  the  efficiency/effectiveness  of  the  Flying  Scheduling 
and  Flying  Execution  processes. 

Strongly  Disagree  Disagree  Neutral  Agree  Strongly  Agree 

o  o  o  o  o 


Please  suggest  any  metrics  that  could  better  depict  efficiency/effectiveness  of  the  Flying  Scheduling  and  Flying  Execution  processes. 
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20.  Please  select  the  metric(s)  that  depict  the  efficiency/effectiveness  of  each  process 
listed. 


Example:  The _ rate  depicts  the  efficiency/effectiveness  of  the  Aircraft 

Process. 

Aircraft  Mx  scheduling 


Mx  Scheduling 


Aircraft  Mx  execution  process 


Abort  Rate  (Air/Ground/Total) 
Aircraft  Availability 


Aircraft  Inventory  (Backup  (BAI)  and  Primary  (PAI)) 
Aircraft  Possessed  (Average,  Primary  Possessed  Rate) 


Average  Sortie  duration  (Actual/Programmed) 
Break  Rate  (C*3  break  rate) 


Cannibalization  Rate 

Career  Development  Course  Success  Rate  (Pass  RateXEnrolled/Completed) 


Delayed  Discrepancy  Rate  (Total.  AWP/AWM,  Avg  per  aircraft) 
Deviation  Rate  (Mx/Ops/Vto,  etc)(Chargeable/Non-Chargeable) 


Fix  Rate  (4/8/12-hour) 

Flying  Schedule  Effectiveness  Rate  (FSE) 


Maintenance  Scheduling  Effectiveness  Rate  (MSE) 


Mission  Capable  Rate  (MC.  FMC.  PMC.  NMC.  TNMC.  M/S/B) 
Mission-Impaired  Capability  Awaiting  Parts  (MICAP) 


Phase/lsochronal  Inspection  Rate  (TDI/Phase  Flow  Average/ISO) 
QA  Pass  Rates  (KTL/PE/RIL/QVI/SI) 


QA  TDV/UCR/DSV  Rates 


Repair  cycle  Time  (Avg  days)  ( Pre/Post-Mx  Time,  Repair  Time) 
Repeat-Recur  Rate  (Repeat  rate.  Recur  Rate) 


Sorties  Scheduled  (Adjusted/Total) 
Spare  Engine  Status 


Supply  Effectiveness  Rate  (Issue/Stockage/Benchstock) 

Training  Rate  (No-Shows/Overdues/Training  Effectiveness  Rate) 


Upgrade  Training  Rate  (by  skill  level)  (school  backlogs) 
UTE  Rate  (Actual/Programmed,  Sortie/Hourly) 


Wbrfccenter  Utilization  Rate 


None  of  the  above 


148 


21.  Please  assess  the  following  statement: 

There  are  adequate  metrics  to  depict  the  efficiency/effectiveness  of  the  Aircraft  Mx 
Scheduling  and  Aircraft  Mx  Execution  processes. 

Strongly  Disagree  Disagree  Neutral  Agree  Strongly  Agree 

o  o  o  o  o 

Please  suggest  any  metrics  that  could  better  depict  efficiency/effectiveness  of  the  Aircraft  Mx  Scheduling  and  Aircraft  Mx  Execution  processes. 
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22.  Please  select  the  metric(s)  that  depict  the  efficiency/effectiveness  of  each  process 
listed. 


Example:  The _ rate  depicts  the  efficiency/effectiveness  of  the  Equipment  Mx 

Scheduling  Process. 


Equipment  Mx  scheduling 

Equipment  Mx  execution 

Abort  Rate  (Air/GroundfTotal) 

□ 

□ 

Aircraft  Availability 

□ 

□ 

Aircraft  Inventory  (Backup  (BAI)  and  Primary  (PAI)) 

□ 

□ 

Aircraft  Possessed  (Average,  Primary  Possessed  Rate) 

□ 

□ 

Average  Sortie  duration  (Actual/Programmed) 

□ 

□ 

Break  Rate  (03  break  rate) 

□ 

□ 

Cannibalization  Rate 

□ 

□ 

Career  Development  Course  Success  Rate  (Pass  RateXEnrolled/Completed) 

□ 

□ 

Delayed  Discrepancy  Rate  (Total.  AWP/AWM.  Avg  per  aircraft) 

□ 

□ 

Deviation  Rate  (Mx/Ops/Vto,  etc)(Chargeable/Non-Chargeable) 

□ 

□ 

Fix  Rate  (4/8/12-hour) 

□ 

□ 

Flying  Schedule  Effectiveness  Rate  (FSE) 

□ 

□ 

Maintenance  Scheduling  Effectiveness  Rate  (MSE) 

□ 

□ 

Manning  Rate 

□ 

□ 

Mission  Capable  Rate  (MC.  FMC.  PMC,  NMC,  TNMC,  M/S/B) 

□ 

□ 

Mission-Impaired  Capability  Awaiting  Parts  (MICAP) 

□ 

□ 

Phase/lsochronal  Inspection  Rate  (TDI/Phase  Flow  Average/ISO) 

□ 

□ 

QA  Pass  Rates  (KTL/PE/RIL/QVI/SI) 

□ 

□ 

QA  TDV/UCR/DSV  Rates 

□ 

□ 

Repair  cycle  Time  (Avg  days)  (Pre/Post-Mx  Time.  Repair  Time) 

□ 

□ 

Repeat-Recur  Rate  (Repeat  rate,  Recur  Rate) 

□ 

□ 

Sorties  Scheduled  (Adjusted/Total) 

□ 

□ 

Spare  Engine  Status 

Supply  Effectiveness  Rate  (Issue/Stockage/Benchstock) 

□ 

□ 

Training  Rate  (No-Shows/Overdues/Training  Effectiveness  Rate) 

□ 

□ 

Upgrade  Training  Rate  (by  skill  level)  (school  backlogs) 

□ 

□ 

UTE  Rate  (Actual/Programmed,  Sortie/Hourly) 

□ 

□ 

Workcenter  Utilization  Rate 

□ 

□ 

None  of  the  above 

□ 

□ 
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23.  Please  assess  the  following  statement: 

There  are  adequate  metrics  to  depict  the  efficiency/effectiveness  of  Equipment  Mx 
Scheduling  and  Equipment  Mx  Execution  processes. 

Strongly  Disagree  Disagree  Neutral  Agree  Strongly  Agree 

o  o  o  o  o 


Please  suggest  metrics  that  could  better  depict  efficiency/effectiveness  of  the  Equipment  Mx  Scheduling  and  Equipment  Mx  Execution 
processes. 
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24.  Please  select  the  metric(s)  that  depict  the  efficiency/effectiveness  of  each  process 
listed. 


Example:  The _ rate  depicts  the  efficiency/effectiveness  of  the  Mx  Training  Scheduling 

Process. 


Mx  Training  Scheduling 

Mx  Training  Execution 

Abort  Rate  (Air/Ground/Total) 

□ 

□ 

Aircraft  Availability 

□ 

□ 

Aircraft  Inventory  (Backup  (BAI)  and  Primary  (PAI)) 

□ 

□ 

Aircraft  Possessed  (Average,  Primary  Possessed  Rate) 

□ 

□ 

Average  Sortie  duration  (Actual/Programmed) 

□ 

□ 

Break  Rate  (03  break  rate) 

□ 

□ 

Cannibalization  Rate 

□ 

□ 

Career  Development  Course  Success  Rate  (Pass  RateXEnrolled/Completed) 

□ 

□ 

Delayed  Discrepancy  Rate  (Total.  AWP/AWM.  Avg  per  aircraft) 

□ 

□ 

Deviation  Rate  (Mx/Ops/Vto,  etc)(Chargeable/Non-Chargeable) 

□ 

□ 

Fix  Rate  (4/8/12-hour) 

□ 

□ 

Flying  Schedule  Effectiveness  Rate  (FSE) 

□ 

□ 

Maintenance  Scheduling  Effectiveness  Rate  (MSE) 

□ 

□ 

Manning  Rate 

□ 

□ 

Mission  Capable  Rate  (MC.  FMC.  PMC,  NMC,  TNMC,  M/S/B) 

□ 

□ 

Mission-Impaired  Capability  Awaiting  Parts  (MICAP) 

□ 

□ 

Phase/lsochronal  Inspection  Rate  (TDI/Phase  Flow  Average/ISO) 

□ 

□ 

QA  Pass  Rates  (KTL/PE/RIL/QVI/SI) 

□ 

□ 

QA  TDV/UCR/DSV  Rates 

□ 

□ 

Repair  cycle  Time  (Avg  days)  (Pre/Post-Mx  Time.  Repair  Time) 

□ 

□ 

Repeat-Recur  Rate  (Repeat  rate,  Recur  Rate) 

□ 

□ 

Sorties  Scheduled  (Adjusted/Total) 

□ 

□ 

Spare  Engine  Status 

Supply  Effectiveness  Rate  (Issue/Stockage/Benchstock) 

□ 

□ 

Training  Rate  (No-Shows/Overdues/Training  Effectiveness  Rate) 

□ 

□ 

Upgrade  Training  Rate  (by  skill  level)  (school  backlogs) 

□ 

□ 

UTE  Rate  (Actual/Programmed,  Sortie/Hourly) 

□ 

□ 

Workcenter  Utilization  Rate 

□ 

□ 

None  of  the  above 

□ 

□ 
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25.  Please  assess  the  following  statement: 

There  are  adequate  metrics  to  depict  the  efficiency/effectiveness  of  Mx  Training 
Scheduling  and  Mx  Training  Execution  processes. 

Strongly  Disagree  Disagree  Neutral  Agree  Strongly  Agree 

o  o  o  o  o 

Please  suggest  metrics  that  could  better  depict  efficiency/effectiveness  of  the  Mx  Training  Scheduling  and  Mx  Training  Execution  processes. 
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26.  Please  select  the  metric(s)  that  depict  the  efficiency/effectiveness  of  each  process 
listed. 

Example:  The _ rate  depicts  the  efficiency/effectiveness  of  the  Aircrew 

Mission/Training  Scheduling  Process. 


Aircrew  Mission/Training  Aircrew  Mission/Training 
scheduling  execution 


Abort  Rate  (Air/Ground/Total) 

□ 

□ 

Aircraft  Availability 

□ 

□ 

Aircraft  Inventory  (Backup  (BAI)  and  Primary  (PAI» 

□ 

□ 

Aircraft  Possessed  (Average,  Primary  Possessed  Rate) 

n 

□ 

Average  Sortie  duration  (Actual/Programmed) 

□ 

□ 

Break  Rate  (C-3  break  rate) 

□ 

□ 

Cannibalization  Rate 

□ 

□ 

Career  Development  Course  Success  Rate  (Pass  RateXEnrolled/Completed) 

□ 

□ 

Delayed  Discrepancy  Rate  (Total.  AWP/AWM,  Avg  per  aircraft) 

□ 

□ 

Deviation  Rate  (Mx/OpsAMc,  etc)(Chargeable/Non-Chargeable) 

□ 

□ 

Fix  Rate  (4/8/12-hour) 

□ 

□ 

Flying  Schedule  Effectiveness  Rate  (FSE) 

□ 

□ 

Maintenance  Scheduling  Effectiveness  Rate  (MSE) 

□ 

□ 

Manning  Rate 

□ 

□ 

Mission  Capable  Rate  (MC.  FMC.  PMC.  NMC.  TNMC.  M/S/B) 

□ 

□ 

Mission-Impaired  Capability  Awaiting  Parts  (MICAP) 

□ 

Phase/lsochronal  Inspection  Rate  (TDI/Phase  Row  Average/ISO) 

□ 

QA  Pass  Rates  (KTL/PE/RIL/QVI/SI) 

□ 

□ 

QA  TDV/UCR/DSV  Rates 

□ 

□ 

Repair  cycle  Time  (Avg  days)  (Pre/Post-Mx  Time.  Repair  Time) 

□ 

□ 

Repeat-Recur  Rate  (Repeat  rate.  Recur  Rate) 

□ 

□ 

Sorties  Scheduled  (Adjusted/Total) 

□ 

□ 

Spare  Engine  Status 

□ 

□ 

Supply  Effectiveness  Rate  (Issue/Stockage/Benchstock) 

□ 

□ 

Training  Rate  (No-Shows/Overdues/Training  Effectiveness  Rate) 

□ 

□ 

Upgrade  Training  Rate  (by  skill  level)  (school  backlogs) 

□ 

□ 

UTE  Rate  (Actual/Programmed.  Sortie/Hourly) 

□ 

□ 

Workcenter  Utilization  Rate 

□ 

□ 

None  of  the  above 

□ 

□ 
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27.  Please  assess  the  following  statement: 

There  are  adequate  metrics  to  depict  the  efficiency/effectiveness  of  Aircrew 
Mission/Training  Scheduling  and  Aircrew  Mission/Training  Execution  processes. 

Strongly  Disagree  Disagree  Neutral  Agree  Strongly  Agree 

o  o  o  o  o 

Please  suggest  metrics  that  could  better  depict  efficiency/effectiveness  of  the  Aircrew  Mission/Training  Scheduling  and  Aircrew  Missions raining 
Execution  processes. 
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The  purpose  of  this  section  is  to  identify  the  metrics  that  best  depict  the  readiness  of  Maintenance  Group  entities- 

28.  For  each  Maintenance  Group  entity  listed  (Maintainers,  Aircraft  and  Equipment),  please 

select  the  metric(s)  that  depicts  the  readiness  of  those  entities. 

Example:  The _ rate  depicts  the  readiness  of  Maintainers,  the  _ 

rate  depicts  the 

readiness  of  Aircraft,  and  the  _  rate  depicts  the  readiness  of  equipment. 

Maintainer 

Equipment 

readiness 

readiness 

Abort  Rate  (Air/Ground/Total) 

□ 

□ 

□ 

Aircraft  Availability 

□ 

□ 

□ 

Aircraft  Inventory  (Backup  (BAI)  and  Primary  (PAI)) 

□ 

□ 

□ 

Aircraft  Possessed  (Average.  Primary  Possessed  Rate) 

n 

□ 

□ 

Average  Sortie  duration  (Actual/Programmed) 

□ 

□ 

□ 

Break  Rate  (C-3  break  rate) 

□ 

□ 

□ 

Cannibalization  Rate 

□ 

□ 

□ 

Career  Development  Course  Success  Rate  (Pass  RateXEnrolled/Completed) 

□ 

□ 

□ 

Delayed  Discrepancy  Rate  (Total.  AWP/AWM.  Avg  per  aircraft) 

□ 

□ 

□ 

Deviation  Rate  (Mx/Ops/Wx.  etc)(Chargeable/Non-Chargeable) 

□ 

□ 

□ 

Fix  Rate  (4/8/12-hour) 

□ 

□ 

□ 

Flying  Schedule  Effectiveness  Rate  (FSE) 

□ 

□ 

□ 

Phase/lsochronal  Inspection  Rate  (TDI/Phase  Row  Average/ISO) 

□ 

□ 

□ 

Maintenance  Scheduling  Effectiveness  Rate  (MSE) 

□ 

□ 

□ 

Mission  Capable  Rate  (MC.  FMC.  PMC.  NMC.  TNMC.  M/S/B) 

□ 

□ 

□ 

Mission-Impaired  Capability  Awaiting  Parts  (MICAP) 

□ 

□ 

□ 

QA  Pass  Rates  (KTL/PE/RIL/QVI/SI) 

□ 

□ 

□ 

QA  TDV/UCR/DSV  Rates 

□ 

□ 

□ 

Repair  cycle  Time  (Avg  days)  (Pre/Post-Mx  Time.  Repair  Time) 

□ 

□ 

□ 

Repeat-Recur  Rate  (Repeat  rate,  Recur  Rate) 

□ 

□ 

□ 

Sorties  Scheduled  (Adjusted/Total) 

□ 

□ 

□ 

Spare  Engine  Status 

□ 

□ 

□ 

Supply  Effectiveness  Rate  (Issue/Stockage/Benchstock) 

□ 

□ 

□ 

Training  Rate  (No-Shows/Overdues/Training  Effectiveness  Rate) 

□ 

□ 

□ 

Upgrade  Training  Rate  (by  skill  level)  (school  backlogs) 

□ 

□ 

□ 

UTE  Rate  (Actual/Programmed.  Sortie/Hourly) 

□ 

□ 

□ 

Workcenter  Utilization  Rate 

□ 

□ 

□ 

Manning  Rate 

□ 

n 

□ 

None  of  the  above 

□ 

□ 

□ 

1 _ 1 
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29.  What  is  your  assessment  of  the  following  statement? 

There  are  adequate  metrics  to  depict  the  readiness  of  Maintenance  Group  entities. 

Strongly  Disagree  Disagree  Neutral  Agree  Strongly  Agree 

o  o  o  o  o 


Please  suggest  metrics  that  could  better  depict  the  readiness  of  Maintenance  Group  entities. 
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Balanced  Scorecard  Perspective  #4:  Workforce  Quality 


The  purpose  of  this  section  is  to  identify  the  metrics  that  best  depict  the  quality  and  skills  of  Maintainers. 

30.  Please  select  the  metric(s)  that  depict  each  of  the  Workforce  Quality  indicators  below. 

Example:  The _ rate  depicts  the  Quality  of  Maintenance  in  the  Maintenance  Group. 

Mx  quality 

Mx  safety 

Abort  Rate  (Air/Ground/Total) 

□ 

□ 

Aircraft  Availability 

□ 

□ 

Aircraft  Inventory  (Backup  (BAI)  and  Primary  (PAI)) 

□ 

11 

Aircraft  Possessed  (Average,  Primary  Possessed  Rate) 

□ 

□ 

Average  Sortie  duration  (Actual/Programmed) 

□ 

1= 

Break  Rate  (C-3  break  rate) 

□ 

□ 

Cannibalization  Rate 

□ 

11 

Career  Development  Course  Success  Rate  (Pass  RateXEnrolled/Completed) 

□ 

□ 

Delayed  Discrepancy  Rate  (Total.  AWP/AWM.  Avg  per  aircraft) 

□ 

c 

Deviation  Rate  (Mx/Ops/Wx.  etc)(Chargeable/Non-Chargeable) 

□ 

c 

Fix  Rate  (4/8/12-hour) 

□ 

□ 

Flying  Schedule  Effectiveness  Rate  (FSE) 

□ 

□ 

Maintenance  Scheduling  Effectiveness  Rate  (MSE) 

□ 

□ 

Manning  Rate 

□ 

□ 

Mission  Capable  Rate  (MC.  FMC.  PMC.  NMC.  TNMC.  M/S/B) 

□ 

□ 

Mission-Impaired  Capability  Awaiting  Parts  (MICAP) 

□ 

□ 

Phase/lsochronal  Inspection  Rate  (TDI/Phase  Flow  Average/ISO) 

□ 

11 

QA  Pass  Rates  (KTL/PE/RIL/QVI/SI) 

□ 

□ 

QA  TDV/UCR/DSV  Rates 

□ 

□ 

Repair  cycle  Time  (Avg  days)  (Pre/Post-Mx  Time.  Repair  Time) 

□ 

□ 

Repeat-Recur  Rate  (Repeat  rate,  Recur  Rate) 

□ 

□ 

Sorties  Scheduled  (Adjusted/Total) 

□ 

□ 

Spare  Engine  Status 

□ 

□ 

Supply  Effectiveness  Rate  (Issue/Stockage/Benchstock) 

□ 

□ 

Training  Rate  (No-Shows/Overdues/Training  Effectiveness  Rate) 

□ 

□ 

Upgrade  Training  Rate  (by  skill  level)  (school  backlogs) 

□ 

□ 

UTE  Rate  (Actual/Programmed.  Sortie/Hourly) 

□ 

11 

Workcenter  Utilization  Rate 

□ 

□ 

None  of  the  above 

□ 

□ 
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31.  Please  select  the  metric(s)  that  depict  each  of  the  Workforce  Quality  indicators  below. 


Example:  The _ rate  depicts  Maintainer  experience  level  in  the  Maintenance  Group. 


Maintainer  skill  level 

Maintainer  experience  level 

Abort  Rate  (Air/Ground/Total) 

□ 

□ 

Aircraft  Availability 

□ 

□ 

Aircraft  Inventory  (Backup  (BAI)  and  Primary  (PAI)) 

□ 

□ 

Aircraft  Possessed  (Average.  Primary  Possessed  Rate) 

□ 

□ 

Average  Sortie  duration  (Actual/Programmed) 

□ 

□ 

Break  Rate  (03  break  rate) 

□ 

□ 

Cannibalization  Rate 

□ 

□ 

Career  Development  Course  Success  Rate  (Pass  RateXEnrolled/Completed) 

□ 

□ 

Delayed  Discrepancy  Rate  (Total.  AWP/AWM,  Avg  per  aircraft) 

□ 

□ 

Deviation  Rate  (Mx/Ops/Wk.  etc)(Chargeable/Non-Chargeable) 

□ 

□ 

Fix  Rate  (4/8/12-hour) 

□ 

□ 

Flying  Schedule  Effectiveness  Rate  (FSE) 

□ 

□ 

Maintenance  Scheduling  Effectiveness  Rate  (MSE) 

□ 

□ 

Manning  Rate 

□ 

□ 

Mission  Capable  Rate  (MC.  FMC.  PMC.  NMC.  TNMC.  M/S/B) 

□ 

□ 

Mission-Impaired  Capability  Awaiting  Parts  (MICAP) 

□ 

□ 

Phase/Isochronal  Inspection  Rate  (TDI/Phase  Flow  Average/ISO) 

□ 

□ 

QA  Pass  Rates  (KTL/PE/RIL/QVI/SI) 

□ 

□ 

QA  TDV/UCR/DSV  Rates 

□ 

□ 

Repair  cycle  Time  (Avg  days)  (Pre/Post-Mx  Time.  Repair  Time) 

□ 

□ 

Repeat-Recur  Rate  (Repeat  rate,  Recur  Rate) 

□ 

□ 

Sorties  Scheduled  (Adjusted/Total) 

□ 

□ 

Spare  Engine  Status 

□ 

□ 

Supply  Effectiveness  Rate  (Issue/Stockage/Benchstock) 

□ 

□ 

Training  Rate  (No-Shows/Overdues/Training  Effectiveness  Rate) 

Upgrade  Training  Rate  (by  skill  level)  (school  backlogs) 

□ 

□ 

UTE  Rate  (Actual/Programmed,  Sortie/Hourly) 

□ 

□ 

Workcenter  Utilization  Rate 

□ 

□ 

None  of  the  above 

□ 

□ 
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32.  Please  select  all  metrics  that  depict  each  of  the  Workforce  Quality  indicators  below, 


Example:  The _ rate  depicts  Maintainer  discipline  in  the  Maintenance  Group. 


Abort  Rate  (Air/Ground/Total) 

Aircraft  Availability 

Aircraft  Inventory  (Backup  (BAI)  and  Primary  (PAI)) 

Aircraft  Possessed  (Average.  Primary  Possessed  Rate) 

Average  Sortie  duration  (Actual/Programmed) 

Break  Rate  (03  break  rate) 

Cannibalization  Rate 

Career  Development  Course  Success  Rate  (Pass  RateXEnrolled/Completed) 
Delayed  Discrepancy  Rate  (Total.  AWP/AWM,  Avg  per  aircraft) 

Deviation  Rate  (Mx/Ops/Wk.  etc)(Chargeable/Non-Chargeable) 

Fix  Rate  (4/8/12-hour) 

Flying  Schedule  Effectiveness  Rate  (FSE) 

Maintenance  Scheduling  Effectiveness  Rate  (MSE) 

Manning  Rate 

Mission  Capable  Rate  (MC.  FMC.  PMC.  NMC.  TNMC.  M/S/B) 
Mission-Impaired  Capability  Awaiting  Parts  (MICAP) 

Phase/Isochronal  Inspection  Rate  (TDI/Phase  Flow  Average/ISO) 

QA  Pass  Rates  (KTL/PE/RIL/QVI/SI) 

QA  TDV/UCR/DSV  Rates 

Repair  cycle  Time  (Avg  days)  (Pre/Post-Mx  Time.  Repair  Time) 
Repeat-Recur  Rate  (Repeat  rate,  Recur  Rate) 

Sorties  Scheduled  (Adjusted/Total) 

Spare  Engine  Status 

Supply  Effectiveness  Rate  (Issue/Stockage/Benchstock) 

Training  Rate  (No-Shows/Overdues/Training  Effectiveness  Rate) 

Upgrade  Training  Rate  (by  skill  level)  (school  backlogs) 

UTE  Rate  (Actual/Programmed,  Sortie/Hourly) 

Workcenter  Utilization  Rate 


Maintainer  Discipline  Maintainer  Retention 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 


None  of  the  above 


□□□□□□□□□□□□□□□□□□□□□□□□□□□DO 


33.  What  is  your  assessment  of  the  following  statement? 

There  are  adequate  metrics  to  depict  the  Workforce  Quality  of  the  Maintenance  Group. 

Strongly  Disagree  Disagree  Neutral  Agree  Strongly  Agree 

o  o  o  o  o 


Please  suggest  metrics  that  could  better  depict  the  Workforce  Quality  of  the  Maintenance  Group. 
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Maintenance  Metric  Set 


The  purpose  of  this  section  is  to  determine  the  utility  of  the  Maintenance  Group  balanced  scorecard,  and  the  optimal 
frequency  of  analysis  of  the  metrics  in  each  perspective 

34.  What  is  your  assessment  of  the  following  statement? 

It  would  be  beneficial  to  study  metrics  representing  each  perspective  (Customer, 
Readiness,  Processes,  Workforce  Quality)  in  the  same  meeting. 

Strongly  Disagree  Disagree  Neutral  Agree  Strongly  Agree 

o  o  o  o  o 


Comments: 
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35.  At  what  management  level,  and  how  often,  should  the  metrics  focusing  on  all 
perspectives  (Customer,  Readiness,  Processes,  Workforce  Quality)  be  studied?  Please 
select  all  that  you  believe  should  apply. 


Never 

daily 

Once  daily 

Weekly 

Monthly 

Quarterly 

Annually 

Flight  Level 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Squadron  Level 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Group  Level 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Wing  Level 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Comments: 
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-□□□□□□□□□□□□□□□□□□□□□□□□□□□ 


Individual  Metrics  Assessment 


The  purpose  of  this  section  of  the  survey  is  to  evaluate  28  individual  maintenance  metrics  based  on  your  expenence 
analyzing/utilizing  the  metric. 

36.  Considering  the  following  maintenance  metrics,  please: 

1)  Indicate  ail  metrics  you  believe  are  NOT  "good"  metrics  forany  reason 
ixplain  your  reason  for  each  metric  in  comment  space  provided 

Abort  Rate  (Air/Ground/Total) 

Aircraft  Availability 

Aircraft  Inventory  (Backup  (BAI)  and  Primary  (PAI)) 

Aircraft  Possessed  (Average.  Primary  Possessed  Rate) 

Average  Sortie  duration  (Acfual/Programmed) 

Break  Rate  (C-3  break  rate) 

Cannibalization  Rate 

Career  Development  Course  Success  Rate  (Pass  Rate)( Enrolled/Completed) 

Delayed  Discrepancy  Rate  (Total,  AWP/AWM,  Avg  per  aircraft) 

Deviation  Rate  (Mx/Ops/ATC/Wk/HHQ/SYM/OTH/SUP/UTEXChargeable/Non-Chargeable) 

Fix  Rate  (4/0/12-hour) 

Flying  Schedule  Effectiveness  Rate  (FSE) 

Isochronal/Phase  Inspection  Rate  (TDI/Phase  Flow  Average/Phase  Inspection  Flow/lSO) 

Maintenance  Scheduling  Effectiveness  Rate  (MSE) 

Mission  Capable  Rate  (MC,  FMC,  PMC.  NMC,  TNMC,  M/S/B) 

Mission-Impaired  Capability  Awaiting  Parts  (MICAP) 

0 A  Pass  Rates  (KTL/PE/RIL/QVI/SI) 

CA  TDVAJCR/DSV  Rates 

Repair  cycle  Time  (Avg  days/Processing  Rate)  (Pre-Mx  Time,  Post-Mx  Time,  Repair  Time) 

Repeat-Recur  Rate  (Repeat  rate.  Recur  Rate) 

Sorties  Scheduled  (Adjusted/Total) 

Spare  Engine  Status 

Supply  Effectiveness  Rate  (Issue/Stockage/Benchstock) 

Training  Rate  (No-Shows/Overdues/Training  Effectiveness  Rate) 

Upgrade  Training  Rate  (by  skill  level)  (school  backlogs) 

UTE  Rate  (Actual/Programmed,  Sortie/Hourly) 

Workcenter  Utilization  Rate 

La - 
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ro  Manning  Rate 


Please  provide  explanations  of  your  above  choices: 
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Appendix  F:  Internal  Review  Board  exemption  letter 


DEPARTMENT  OF  THE  AIR  FORCE 
AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 
WRIGHT-PATTERSON  AIR  FORCE  BASE  OHIO 


1 8  December  20 1 3 


MEMORANDUM  FOR  LT  COL  JOSEPH  HUSCROFT 

FROM:  Joseph  B.  Skipper,  Lt  Col,  Ph.D. 

AFIT  IRB  Research  Reviewer 
2950  Hobson  Way 

Wright-Patterson  AFB,  OH  45433-7765 

SUBJECT:  Approval  for  exemption  request  from  human  experimentation  requirements  (32  CFR 
2 1 9,  DoDD  32 1 6.2  and  AFI  40-402)  for  “An  Analysis  of  Aircraft  Maintenance  Metrics.” 

1.  Your  request  was  based  on  the  Code  of  Federal  Regulations,  title  32,  part  219,  section  101, 
paragraph  (b)  (2)  Research  activities  that  involve  the  use  of  educational  tests  (cognitive, 
diagnostic,  aptitude,  achievement),  survey  procedures,  interview  procedures,  or  observation  of 
public  behavior  unless:  (i)  Information  obtained  is  recorded  in  such  a  manner  that  human 
subjects  can  be  identified,  directly  or  through  identifiers  linked  to  the  subjects;  and  (ii)  Any 
disclosure  of  the  human  subjects’  responses  outside  the  research  could  reasonably  place  the 
subjects  at  risk  of  criminal  or  civil  liability  or  be  damaging  to  the  subjects’  financial  standing, 
employability,  or  reputation. 

2.  Your  study  qualifies  for  this  exemption  because  you  are  not  collecting  sensitive  data,  which 
could  reasonably  damage  the  subjects’  financial  standing,  employability,  or  reputation.  Further, 
the  demographic  data  you  are  collecting  cannot  realistically  be  expected  to  map  a  given  response 
to  a  specific  subject.  Your  plan  includes  ample  and  appropriate  measures  to  safeguard  any 
information  collected  and  your  mitigation  plan  should  such  breech  occur  is  adequate. 

3.  This  determination  pertains  only  to  the  Federal,  Department  of  Defense,  and  Air  Force 
regulations  that  govern  the  use  of  human  subjects  in  research.  Further,  if  a  subject’s  future 
response  reasonably  places  them  at  risk  of  criminal  or  civil  liability  or  is  damaging  to  their 
financial  standing,  employability,  or  reputation,  you  are  required  to  file  an  adverse  event  report 
with  this  office  immediately. 


Appendix  G.  Survey  Control  Number 


5  February  2014 

MEMORANDUM  FOR  AIR  FORCE  INSTITUTE  OF  TECHNOLOGY  (AFIT) 
ATTENTION:  LT  COL  JOSEPH  HUSCROFT 

FROM:  AFPC/DSYS 

550  C  Street  West 
Randolph  AFBTX  78158 

SUBJECT:  Survey  Approval  —  Maintenance  Metrics  Survey. 

1 .  The  survey  is  approved  for  use  with  the  following  population(s): 


Population: 

Number(s): 

Air  Force  Active-Duty'  Officers 

1.500 

Air  Force  Active-Duty  Enlisted 

0 

Air  Force  Civilians 

0 

Air  Force  Retirees  and  or  AF  Family  Members 

0 

Total  Number  to  be  Surveyed 

1.500 

The  Survey  Control  Number  (SCX)  for  this  effort  is  AF14-123  AFIT  This  SCN  is  valid  from 
02/14/2014  through  2  28  2014. 

Please  ensure  compliance  with  the  following  guidance,  as  applicable,  while  administering  your 
survey. 

a.  Invitations  to  participate  in  the  survey  must  include: 

(1)  Survey'  title  (as  shown  m  the  subject  line  of  this  memo). 

(2)  AF  Survey  Control  Number  (SCN). 

(3)  Statement  that  completion  of  the  survey  is  voluntary. 

(4)  Link  to  the  list  of  Air  Force  approved  survey's:  https  www.mv  af  mil  gcss 

af  LTSAF  ep  brow'se.do?prograinTd=t?D8EB9D6297405FA012980243147010A&ehannelPage 

Id=s5FDEA9F02134FFA70121351677C8004S. 

(5)  Government  contact  name  or  office,  with  official  contact  information  (e.g.,  e-mail  address, 
telephone  number,  etc.),  to  provide  a  point  of  contact  for  questions  about  the  survey. 

(6)  Identifying  information  of  the  survey’s  sponsor,  to  inform  survey  recipients  under  w'hose 
authority'  the  survey  is  being  conducted. 

(7)  All  AF  attitude  and  opinion  surveys  must  include  the  follow'ing  statement  on  the 
questionnaire:  "We  cannot  provide  confidentiality'  to  a  participant  regarding  comments 
involving  criminal  activity/behavior,  or  statements  that  pose  a  threat  to  yourself  or  others. 

Do  NOT  discuss  or  comment  on  classified  or  operationally  sensitive  information." 
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b.  This  approval  is  exclusive  to  the  Air  Force  community  and  does  not  constitute  authority  for 
administration  to  individuals  from  other  federal  agencies,  sister  services,  etc.  Surveys  that 
include  individuals  from  outside  the  Air  Force  community  must  be  coordinated  through  the 
DOD'WHS/ESCD  Information  Management  Division  (commercial  phone  703-696-5284). 

c.  Tlie  organization  conducting  this  survey  must  contact  the  Civilian  Personnel  Office;  Civilian 
Personnel  Element,  Manpower  &  Personnel  Flight;  for  labor  union  notification  prior  to  releasing  tins 
survey  if  any  participants  are  civilian  employees  of  a  bargaining  unit.  If  this  survey  involves 
bargaining  unit  civilians  at  more  than  one  base,  the  organization  conducting  this  survey  must  notify 
HQ  AFPC/DPEECC.  Air  Force  Program  Management  and  Evaluation. 

d.  The  organization  conducting  this  survey  must  insure  that  if  this  survey  requires  any  changes, 
request  must  be  submitted  to  the  Survey  Office  for  review'  and  approval  prior  to  implementation 
in  accordance  with  AFI 38-501. 

e.  This  survey  does/does  not  require  review'  by  an  Air  Force  Institutional  Review'  Board.  If  this 
survey  requires  an  IRB.  the  PI  must  submit  all  proposed  survey  changes  to  the  Survey  and  IRB 
Office  for  review  and  approval  (nunor  changes  do  not  require  a  change  of  SCN  number)  prior  to 
implementation  in  accordance  with  AFI  38-501. 

f.  AFI  33-129,  Web  Management  and  Internet  Use.  paragraph  3.2.5.;  3.7.4,  and  3.7.5;  Please 
contact  SAF/XCDIG.  1800  Air  Force  Pentagon.  Washington  DC  20330-1800.  for  further 
guidance  details  which  requires  that  all  websites  hosted  in  the  commercial  environment  (i.e., 

“  com",  “.org”,  etc.)  receive  SAF/A6  approval.  The  organization  conducting  tlus  survey  must 
coordinate  with  SAF/A6  (e-mail  address  A3CS.A6CSStrategy(5)pentagon.af.mil)  for  approval  of  a 
w'aiver  if  the  survey  will  be  hosted  on  any  website  other  than  a  “.mil"  account.  If  a  w'aiver  is 
required,  it  must  be  granted  by  SAF/A6  prior  to  administration  of  the  survey. 

g.  For  information  regarding  digital  certification  of  e-maiLs.  refer  to  AFI  33-1 19,  Air  Force 
Messaging.  The  reference  for  PK  enabling  (PKE)  information  is 

https:/7afpki  lackland.af.mil  html  pkenabling.cfm.  For  mfonnation  pertaining  to  “.mil"  accounts, 
the  reference  is  https  afpki. lackland  af  mil  html  help  desk.cfm.  Information  for  systems  that 
are  not  “.mil”  can  be  found  at  http: '  iase. disa. mil  pki  eca1.  For  information  on  External 
Certificate  Authority  or  to  contact  a  representative,  the  reference  is 
http://iase.disa.mil  pki/eca/contact  us.html. 

h.  The  organization  conducting  this  survey  must  ensure  its  Operations  Security  (OPSEC) 
manager  review's  this  survey  prior  to  administration.  References  for  the  OPSEC  Program 
include:  DOD  Directive  5205.02,  DOD  Operations  Security  Program:  Joint  Publication  3-13.3, 
Operations  Security ;  AFPD  10-7,  Air  Force  Information  Operations:  and  AFI  10-701, 
Operations  Security  (OPSEC). 

l.  The  public  may  request  survey  results  under  provisions  of  the  Freedom  of  Information  Act 
(FOIA).  Results  released  outside  the  Air  Force  require  coordination  w'lth  Air  Force  Public 
Affairs  prior  to  dissemination. 
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j.  Data  collected  under  this  survey  may  be  subject  to  the  Privacy  Act  of  1974.  Please  ensure 
compliance  with  this  act  as  set  forth  in  Title  5  United  States  Code  (USC),  Sec  552a;  Title  10 
USC,  Sec  55  and  8013;  Executive  Order  9397:  and  Air  Force  Instruction  33-332,  Privacy  Act 
Program. 

2.  If  you  have  any  questions,  please  call  the  Air  Force  Survey  Office  at  DSN  487-5332  or  send  an 
e-mail  to  af.survev  7/  us  af  mil 


//Signed// 

RENEE  TEALER 
Management  Analyst 
Air  Force  Survey  Office 
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Appendix  H:  Process  and  Assigned  Metrics  Cross  Tabulation 
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Appendix  I:  Perspective  Category  Correlation  Coefficients 
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Aircrew  Mission/  Training  execution 
Mx  Training  Scheduling 
Flying  Scheduling  Process 
Aircrew  Mission/  Training  scheduling 
Equipment  readiness 


Aircraft  Mx  scheduling  process 
Equipment  Mx  execution 


Aircraft  Mx  scheduling  process 
Equipment  Mx  scheduling 


Equipment  readiness 
Aircrew  Mission/  Training  scheduling 
Aircrew  Mission/  Training  scheduling 
Flying  Scheduling  Process 
Aircraft  readiness 
Aircraft  Mx  scheduling  process 
Aircraft  Mx  scheduling  process 
Flying  Execution  Process 


-0.0856 

-0.0924 

-0.0959 

-0.1084 

-0.1137 

-0.1265 

-0.1283 

-0.1382 

-0.1416 

-0.1588 

-0.1619 


-0.1628 

-0.1674 


-0.1783 

-0.1784 


-0.1796 

-0.1899 

-0.1922 

-0.194 

-0.2037 

-0.2103 

-0.2178 

-0.2307 

-0.235 

-0.2426 

-0.2494 

-0.2506 

-0.2515 


-0.2548 

-0.2582 


-0.2644 

-0.2777 


-0.2916 

-0.2975 

-0.3012 

-0.3069 

-0.3118 

-0.3152 

-0.3269 

-0.3343 
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Aircrew  Mission/  Training  scheduling 

Mx  Training  Execution 

-0.3351 

Maintainer  experience  level 

Flying  Scheduling  Process 

-0.3357 

Maintainer  Retention 

Aircrew  Mission/  Training  scheduling 

-0.3447 

Maintainer  Retention 

Aircraft  Mx  scheduling  process 

-0.3467 

Maintainer  Retention 

Flying  Scheduling  Process 

-0.3661 

Maintainer  Discipline 

Flying  Scheduling  Process 

-0.3721 

Maintainer  skill  level 

Aircrew  Mission/  Training  scheduling 

-0.3743 

Maintainer  Discipline 

Aircrew  Mission/  Training  scheduling 

-0.3901 

Aircrew  Mission/  Training  scheduling 

Equipment  Mx  execution 

-0.3925 

Mx  safety 

Aircraft  Mx  scheduling  process 

-0.3936 

Mx  Training  Execution 

Flying  Scheduling  Process 

-0.3937 

Mx  safety 

Flying  Scheduling  Process 

-0.3942 

Maintainer  skill  level 

Flying  Scheduling  Process 

-0.4349 

Equipment  readiness 

Mx  Training  Scheduling 

-0.462 
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Appendix  J:  Thesis  Sponsorship  Letter 


DEPARTMENT  OF  THE  AIR  FORCE 

HEADQUARTERS  UNITED  STATES  AIR  FORCE 
WASHINGTON,  DC 


MEMORANDUM  FOR  Air  Force  Institute  of  Technology  (AFIT)  Survey  Control  Panel 


HQ  AFPC  MAPP 


FROM:  AF/A4L 

SUBJECT:  AFIT  Student  Thesis  Sponsorship:  Maintenance  Metrics  Survey 

This  is  to  inform  you  that  Major  Adrienne  L.  Stahl,  an  AFIT  graduate  student,  is 
conducting  a  study  for  my  office  that  will  evaluate  Aircraft  Maintenance  Performance  Metrics, 
both  individually  and  as  a  set.  First,  she  will  seek  to  determine  if  the  current  metrics  set  is 
sufficiently  aligned  with  the  strategic  objectives  of  a  CAF  flying  wing.  In  addition.  Major  Stahl 
will  perform  a  review  of  the  aircraft  maintenance  performance  measurement  system  and 
individual  metrics  to  evaluate  them  against  a  set  of  objective  evaluation  criteria.  Lastly.  Major 
Stalil  will  review  the  frequency  of  mandated  maintenance  metrics  study,  and  determine  the 
optimal  interval  of  meetings  for  the  purpose  of  the  analysis  of  aircraft  maintenance  metrics  study 
amongst  group  and  wing  level  key  players.  Major  Stahl  will  administer  a  web-based  survey  to 
collect  data  from  Ah  Force  Maintenance  Officers. 

Determination  of  the  most  optimal  maintenance  metrics  set.  made  up  of  strategically 
aligned  metrics  that  are  studied  at  an  optimal  frequency  will  assist  maintenance  leaders  and 
decision  makers  in  evaluating  the  performance  of  their  aircraft  and  maintenance  operations,  their 
impact  on  flying  operations,  and  the  readiness  of  the  wing. 

If  you  have  any  questions,  please  contact  Lt  Col  Joseph  Huscroft  -  Phone  937-255-3636. 
ext.  4538:  E-mail  —  joseph.huscroft@afit.edu. 


CT-OOOffc*  -KJMNB  wnt/w ! 
_«•  2014  0204  14  m  4*  -OOW 


..*•  2014  0204  14  m  4»-0i1» 


JOHN  B.  COOPER.  Maj  Gen.  USAF 
Director  of  Logistics 

DCS  Logistics.  Installations  &  Mission  Support 


176 


Appendix  K:  Metrics  Proposal  Comparison 


Metrics 

Stahl 

Waller 

Flarris 

Nachtman  et  al 

Abort  Rate  (Air/Ground/Total) 

Incorporate 

Remove 

n/a 

Incorporate 

Aircraft  Availability 

Incorporate 

n/a 

n/a 

n/a 

Aircraft  Inventory  (Backup  (BAI)  and  Primary  (PAI)) 

Eliminate 

Re-evaluate 

n/a 

n/a 

Aircraft  Possessed  (Average,  Primary  Possessed  Rate) 

Eliminate 

Re-evaluate 

n/a 

n/a 

Average  Sortie  duration  (Actual/Programmed) 

Incorporate 

Re-evaluate 

n/a 

n/a 

Break  Rate  (C-3  break  rate) 

Incorporate 

Re-evaluate 

n/a 

n/a 

Cannibalization  Rate 

Re-evaluate 

Remove 

n/a 

Incorporate 

Career  Development  Course  Success  Rate  (Pass 
Rate)(Enrolled/Completed) 

Incorporate 

n/a 

n/a 

n/a 

Delayed  Discrepancy  Rate  (Total,  AWP/AWM,  Avg  per 
aircraft) 

Incorporate 

Incorporate 

n/a 

Incorporate 

Deviation  Rate 

( Mx/Ops/ ATC/Wx/H  HQ/SYM/OTH/SU  P/UTE, 
Chargeable/Non-Chargeable) 

Incorporate 

Re-evaluate 

n/a 

Incorporate 

Fix  Rate  (4/8/12-hr) 

Eliminate 

Incorporate 

n/a 

Incorporate 

Flying  Schedule  Effectiveness  rate  (FSE) 

Incorporate 

Re-evaluate 

n/a 

Incorporate 

Maintenance  Scheduling  Effectiveness  Rate  (MSE) 

Incorporate 

Incorporate 

n/a 

Incorporate 

Manning  Rate 

Incorporate 

n/a 

n/a 

n/a 

Mission  Capable  Kate  (ML,  fml,  PMC,  nml,  inmc, 

M/S/B) 

Incorporate 

n/a 

n/a 

Mission-Impaired  Capability  Awaiting  Parts  (MICAP) 

Re-evaluate 

Incorporate 

n/a 

Incorporate 

Phase/lsochronal  Inspection  Rate  (TDI/Phase  Flow 
Average/Phase  Inspection  Flow/ISO) 

Incorporate 

n/a 

n/a 

Incorporate 

QA  Pass  Rates  (KTL/PE/RIL/QVI/SI) 

Incorporate 

n/a 

n/a 

n/a 

QA  TDV/UCR/DSV  Rates 

Incorporate 

n/a 

n/a 

n/a 

Repair  cycle  Time  (Avg  days/Processing  Rate)  (Pre-Mx 
Time,  Post-MxTime,  Repair  Time)  (Base  Repair  Cycle 

Time)  (Overall  Base  Repair  Cycle  Processing  Time) 

Incorporate 

n/a 

n/a 

n/a 

Repeat-Recur  Rate  (Repeat  rate.  Recur  Rate) 

Incorporate 

Incorporate 

n/a 

Incorporate 

Sorties  Scheduled  (Adjusted/Total) 

Incorporate 

n/a 

n/a 

n/a 

Spare  engine  Status 

Incorporate 

n/a 

n/a 

n/a 

Supply  Effectiveness  Rate  (Issue/Stockage/Benchstock) 

Re-evaluate 

n/a 

n/a 

n/a 

Training  Rate  (No-Shows/Overdues/Training 

Effectiveness  Rate) 

Incorporate 

n/a 

n/a 

Incorporate 

Upgrade  Training  rate  (by  skill  level)  (school  backlogs) 

Incorporate 

n/a 

n/a 

Incorporate 

UTE  Rate  (Actual/Programmed)  (Sortie/Hourly) 

Incorporate 

Re-evaluate 

n/a 

n/a 

Workcenter  Utilization  Rate 

Re-evaluate 

n/a 

n/a 

n/a 

Job  Standard  Fix  Rate 

Create 

n/a 

n/a 

n/a 

Equipment  Maintenance  Scheduling  Effectiveness  Rate 

Create 

n/a 

Create 

n/a 

Equipment  Effectiveness/Break  Rate 

Create 

n/a 

n/a 

n/a 

Equipment  Fix  Rate 

Create 

n/a 

n/a 

n/a 

Equipment  MICAP  Rate 

Create 

n/a 

Create 

n/a 

Maintainer  Availability 

Create 

Incorporate 

n/a 

n/a 

Maintainer  Capability  Rate 

Create 

Incorporate 

n/a 

n/a 

Equipment  MC/ln-Commission  Rate 

Create 

n/a 

Create 

n/a 

Equipment  Availability 

Create 

n/a 

n/a 

n/a 

Mission  Effectiveness  Rate 

Create 

n/a 

n/a 

n/a 

Operations  Scheduling  Effectiveness  Rate 

Create 

n/a 

n/a 

n/a 

RAP  Milestones  Rate 

Create 

n/a 

n/a 

n/a 

Maintainer  Experience  Level 

Create 

n/a 

n/a 

n/a 

Maintainer  Skill  Level 

Create 

n/a 

n/a 

Incorporate 

Overtime  Rate 

Create 

n/a 

n/a 

n/a 

Weekend  Duty  Rate 

Create 

n/a 

n/a 

n/a 

Re-enlistment  Rate 

Create 

n/a 

n/a 

n/a 
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